Skip to content

Overview

Splunk Solutions SDK is an open source packaged solution for getting data into Splunk using modular inputs. This SDK is used by Splunk Add-on builder, and Splunk UCC based add-ons and is intended for use by partner developers. This SDK/Library extends the Splunk SDK for Python.

Note: this project uses poetry 1.5.1.

Removed requests and urllib3 from solnlib

The requests and urllib3 libraries has been removed from solnlib, so solnlib now depends on the requests and urllib3 libraries from the running environment. By default, Splunk delivers the above libraries and their version depends on the Splunk version. More information here.

IMPORTANT: urllib3 is available in Splunk v8.1.0 and later

Please note that if requests or urllib3 are installed in <Add-on>/lib e.g. as a dependency of another library, that version will be taken first. If requests or urllib3 is missing in the add-on’s lib directory, the version provided by Splunk will be used.

Custom Version of requests and urllib3

In case the Splunk’s requests or urllib3 version is not sufficient for you, you can deliver version you need by simply adding it to the requirements.txt or pyproject.toml file in your add-on.

Use solnlib outside the Splunk

Solnlib no longer provides requests and urllib3 so if you want to use solnlib outside the Splunk, please note that you will need to provide these libraries yourself in the environment where solnlib is used.

References ↵

checkpointer.py

This module provides two kinds of checkpointer: KVStoreCheckpointer, FileCheckpointer for modular input to save checkpoint.

__all__ = ['CheckpointerException', 'KVStoreCheckpointer', 'FileCheckpointer'] module-attribute

Checkpointer

Base class of checkpointer.

Source code in solnlib/modular_input/checkpointer.py 
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
class Checkpointer(metaclass=ABCMeta):
    """Base class of checkpointer."""

    @abstractmethod
    def update(self, key: str, state: Any):
        """Updates document with an id that equals to `key` and `state` as
        document data."""

    @abstractmethod
    def batch_update(self, states: Iterable[Dict[str, Any]]):
        """Updates multiple documents."""

    @abstractmethod
    def get(self, key: str) -> dict:
        """Gets document with an id that equals to `key`."""

    @abstractmethod
    def delete(self, key: str):
        """Deletes document with an id that equals to `key`."""

batch_update(states) abstractmethod

Updates multiple documents.

Source code in solnlib/modular_input/checkpointer.py 
49
50
51
@abstractmethod
def batch_update(self, states: Iterable[Dict[str, Any]]):
    """Updates multiple documents."""

delete(key) abstractmethod

Deletes document with an id that equals to key.

Source code in solnlib/modular_input/checkpointer.py 
57
58
59
@abstractmethod
def delete(self, key: str):
    """Deletes document with an id that equals to `key`."""

get(key) abstractmethod

Gets document with an id that equals to key.

Source code in solnlib/modular_input/checkpointer.py 
53
54
55
@abstractmethod
def get(self, key: str) -> dict:
    """Gets document with an id that equals to `key`."""

update(key, state) abstractmethod

Updates document with an id that equals to key and state as document data.

Source code in solnlib/modular_input/checkpointer.py 
44
45
46
47
@abstractmethod
def update(self, key: str, state: Any):
    """Updates document with an id that equals to `key` and `state` as
    document data."""

CheckpointerException

Bases: Exception

Source code in solnlib/modular_input/checkpointer.py 
37
38
class CheckpointerException(Exception):
    pass

FileCheckpointer

Bases: Checkpointer

File checkpointer.

Use file to save modular input checkpoint.

Examples:

>>> from solnlib.modular_input import checkpointer
>>> ck = checkpointer.FileCheckpointer('/opt/splunk/var/...')
>>> ck.update(...)
>>> ck.get(...)
Source code in solnlib/modular_input/checkpointer.py 
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
class FileCheckpointer(Checkpointer):
    """File checkpointer.

    Use file to save modular input checkpoint.

    Examples:
        >>> from solnlib.modular_input import checkpointer
        >>> ck = checkpointer.FileCheckpointer('/opt/splunk/var/...')
        >>> ck.update(...)
        >>> ck.get(...)
    """

    def __init__(self, checkpoint_dir: str):
        """Initializes FileCheckpointer.

        Arguments:
            checkpoint_dir: Checkpoint directory.
        """
        warnings.warn(
            "FileCheckpointer is deprecated, please use KVStoreCheckpointer",
            stacklevel=2,
        )
        self._checkpoint_dir = checkpoint_dir

    def encode_key(self, key):
        return base64.b64encode(key.encode()).decode()

    def update(self, key, state):
        file_name = op.join(self._checkpoint_dir, self.encode_key(key))
        with open(file_name + "_new", "w") as fp:
            json.dump(state, fp)

        if op.exists(file_name):
            try:
                os.remove(file_name)
            except OSError:
                pass

        os.rename(file_name + "_new", file_name)

    def batch_update(self, states):
        for state in states:
            self.update(state["_key"], state["state"])

    def get(self, key):
        file_name = op.join(self._checkpoint_dir, self.encode_key(key))
        try:
            with open(file_name) as fp:
                return json.load(fp)
        except (OSError, ValueError):
            return None

    def delete(self, key):
        file_name = op.join(self._checkpoint_dir, self.encode_key(key))
        try:
            os.remove(file_name)
        except OSError:
            pass

__init__(checkpoint_dir)

Initializes FileCheckpointer.

Parameters:

Name Type Description Default
checkpoint_dir str

Checkpoint directory.

 required
Source code in solnlib/modular_input/checkpointer.py 
218
219
220
221
222
223
224
225
226
227
228
def __init__(self, checkpoint_dir: str):
    """Initializes FileCheckpointer.

    Arguments:
        checkpoint_dir: Checkpoint directory.
    """
    warnings.warn(
        "FileCheckpointer is deprecated, please use KVStoreCheckpointer",
        stacklevel=2,
    )
    self._checkpoint_dir = checkpoint_dir

batch_update(states)

Source code in solnlib/modular_input/checkpointer.py 
246
247
248
def batch_update(self, states):
    for state in states:
        self.update(state["_key"], state["state"])

delete(key)

Source code in solnlib/modular_input/checkpointer.py 
258
259
260
261
262
263
def delete(self, key):
    file_name = op.join(self._checkpoint_dir, self.encode_key(key))
    try:
        os.remove(file_name)
    except OSError:
        pass

encode_key(key)

Source code in solnlib/modular_input/checkpointer.py 
230
231
def encode_key(self, key):
    return base64.b64encode(key.encode()).decode()

get(key)

Source code in solnlib/modular_input/checkpointer.py 
250
251
252
253
254
255
256
def get(self, key):
    file_name = op.join(self._checkpoint_dir, self.encode_key(key))
    try:
        with open(file_name) as fp:
            return json.load(fp)
    except (OSError, ValueError):
        return None

update(key, state)

Source code in solnlib/modular_input/checkpointer.py 
233
234
235
236
237
238
239
240
241
242
243
244
def update(self, key, state):
    file_name = op.join(self._checkpoint_dir, self.encode_key(key))
    with open(file_name + "_new", "w") as fp:
        json.dump(state, fp)

    if op.exists(file_name):
        try:
            os.remove(file_name)
        except OSError:
            pass

    os.rename(file_name + "_new", file_name)

KVStoreCheckpointer

Bases: Checkpointer

KVStore checkpointer.

Use KVStore to save modular input checkpoint.

More information about KV Store in Splunk is here.

Examples:

>>> from solnlib.modular_input import checkpointer
>>> checkpoint = checkpointer.KVStoreCheckpointer(
        "unique_addon_checkpoints",
        "session_key",
        "unique_addon"
    )
>>> checkpoint.update("input_1", {"timestamp": 1638043093})
>>> checkpoint.get("input_1")
>>> # returns {"timestamp": 1638043093}
Source code in solnlib/modular_input/checkpointer.py 
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
class KVStoreCheckpointer(Checkpointer):
    """KVStore checkpointer.

    Use KVStore to save modular input checkpoint.

    More information about KV Store in Splunk is
    [here](https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/aboutkvstorecollections).

    Examples:
        >>> from solnlib.modular_input import checkpointer
        >>> checkpoint = checkpointer.KVStoreCheckpointer(
                "unique_addon_checkpoints",
                "session_key",
                "unique_addon"
            )
        >>> checkpoint.update("input_1", {"timestamp": 1638043093})
        >>> checkpoint.get("input_1")
        >>> # returns {"timestamp": 1638043093}
    """

    def __init__(
        self,
        collection_name: str,
        session_key: str,
        app: str,
        owner: Optional[str] = "nobody",
        scheme: Optional[str] = None,
        host: Optional[str] = None,
        port: Optional[int] = None,
        **context: Any,
    ):
        """Initializes KVStoreCheckpointer.

        Arguments:
            collection_name: Collection name of kvstore checkpointer.
            session_key: Splunk access token.
            app: App name of namespace.
            owner: (optional) Owner of namespace, default is `nobody`.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            context: Other configurations for Splunk rest client.

        Raises:
            binding.HTTPError: HTTP error different from 404, for example 503
                when KV Store is initializing and not ready to serve requests.
            CheckpointerException: If init KV Store checkpointer failed.
        """
        try:
            if not context.get("pool_connections"):
                context["pool_connections"] = 5
            if not context.get("pool_maxsize"):
                context["pool_maxsize"] = 5
            self._collection_data = _utils.get_collection_data(
                collection_name,
                session_key,
                app,
                owner,
                scheme,
                host,
                port,
                {"state": "string"},
                **context,
            )
        except KeyError:
            raise CheckpointerException("Get KV Store checkpointer failed.")

    @utils.retry(exceptions=[binding.HTTPError])
    def update(self, key: str, state: Any) -> None:
        """Updates document with an id that equals to `key` and `state` as
        document data.

        Arguments:
            key: `id` of the document to update.
            state: Document data to update. It can be integer, string,
                or a dict, or anything that can be an argument to `json.dumps`.

        Raises:
            binding.HTTPError: when an error occurred in Splunk, for example,
                when Splunk is restarting and KV Store is not yet initialized.
        """
        record = {"_key": key, "state": json.dumps(state)}
        self._collection_data.batch_save(record)

    @utils.retry(exceptions=[binding.HTTPError])
    def batch_update(self, states: Iterable[Dict[str, Any]]) -> None:
        """Updates multiple documents.

        Arguments:
            states: Iterable that contains documents to update. Document should
                be a dict with at least "state" key.

        Raises:
            binding.HTTPError: when an error occurred in Splunk, for example,
                when Splunk is restarting and KV Store is not yet initialized.
        """
        for state in states:
            state["state"] = json.dumps(state["state"])
        self._collection_data.batch_save(*states)

    @utils.retry(exceptions=[binding.HTTPError])
    def get(self, key: str) -> Optional[Any]:
        """Gets document with an id that equals to `key`.

        Arguments:
            key: `id` of the document to get.

        Raises:
            binding.HTTPError: When an error occurred in Splunk (not 404 code),
                can be 503 code, when Splunk is restarting and KV Store is not
                yet initialized.

        Returns:
            Document data under `key` or `None` in case of no data.
        """
        try:
            record = self._collection_data.query_by_id(key)
        except binding.HTTPError as e:
            if e.status != 404:
                logging.error(f"Get checkpoint failed: {traceback.format_exc()}.")
                raise
            return None
        return json.loads(record["state"])

    @utils.retry(exceptions=[binding.HTTPError])
    def delete(self, key: str) -> None:
        """Deletes document with an id that equals to `key`.

        Arguments:
            key: `id` of the document to delete.

        Raises:
            binding.HTTPError: When an error occurred in Splunk (not 404 code),
                can be 503 code, when Splunk is restarting and KV Store is not
                yet initialized.
        """
        try:
            self._collection_data.delete_by_id(key)
        except binding.HTTPError as e:
            if e.status != 404:
                logging.error(f"Delete checkpoint failed: {traceback.format_exc()}.")
                raise

__init__(collection_name, session_key, app, owner='nobody', scheme=None, host=None, port=None, **context)

Initializes KVStoreCheckpointer.

Parameters:

Name Type Description Default
collection_name str

Collection name of kvstore checkpointer.

 required
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner Optional[str]

(optional) Owner of namespace, default is nobody.

 'nobody'
scheme Optional[str]

(optional) The access scheme, default is None.

 None
host Optional[str]

(optional) The host name, default is None.

 None
port Optional[int]

(optional) The port number, default is None.

 None
context Any

Other configurations for Splunk rest client.

 {}

Raises:

Type Description
binding.HTTPError

HTTP error different from 404, for example 503 when KV Store is initializing and not ready to serve requests.
CheckpointerException

If init KV Store checkpointer failed.
Source code in solnlib/modular_input/checkpointer.py 
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
def __init__(
    self,
    collection_name: str,
    session_key: str,
    app: str,
    owner: Optional[str] = "nobody",
    scheme: Optional[str] = None,
    host: Optional[str] = None,
    port: Optional[int] = None,
    **context: Any,
):
    """Initializes KVStoreCheckpointer.

    Arguments:
        collection_name: Collection name of kvstore checkpointer.
        session_key: Splunk access token.
        app: App name of namespace.
        owner: (optional) Owner of namespace, default is `nobody`.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Raises:
        binding.HTTPError: HTTP error different from 404, for example 503
            when KV Store is initializing and not ready to serve requests.
        CheckpointerException: If init KV Store checkpointer failed.
    """
    try:
        if not context.get("pool_connections"):
            context["pool_connections"] = 5
        if not context.get("pool_maxsize"):
            context["pool_maxsize"] = 5
        self._collection_data = _utils.get_collection_data(
            collection_name,
            session_key,
            app,
            owner,
            scheme,
            host,
            port,
            {"state": "string"},
            **context,
        )
    except KeyError:
        raise CheckpointerException("Get KV Store checkpointer failed.")

batch_update(states)

Updates multiple documents.

Parameters:

Name Type Description Default
states Iterable[Dict[str, Any]]

Iterable that contains documents to update. Document should be a dict with at least “state” key.

 required

Raises:

Type Description
binding.HTTPError

when an error occurred in Splunk, for example, when Splunk is restarting and KV Store is not yet initialized.
Source code in solnlib/modular_input/checkpointer.py 
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
@utils.retry(exceptions=[binding.HTTPError])
def batch_update(self, states: Iterable[Dict[str, Any]]) -> None:
    """Updates multiple documents.

    Arguments:
        states: Iterable that contains documents to update. Document should
            be a dict with at least "state" key.

    Raises:
        binding.HTTPError: when an error occurred in Splunk, for example,
            when Splunk is restarting and KV Store is not yet initialized.
    """
    for state in states:
        state["state"] = json.dumps(state["state"])
    self._collection_data.batch_save(*states)

delete(key)

Deletes document with an id that equals to key.

Parameters:

Name Type Description Default
key str

id of the document to delete.

 required

Raises:

Type Description
binding.HTTPError

When an error occurred in Splunk (not 404 code), can be 503 code, when Splunk is restarting and KV Store is not yet initialized.
Source code in solnlib/modular_input/checkpointer.py 
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
@utils.retry(exceptions=[binding.HTTPError])
def delete(self, key: str) -> None:
    """Deletes document with an id that equals to `key`.

    Arguments:
        key: `id` of the document to delete.

    Raises:
        binding.HTTPError: When an error occurred in Splunk (not 404 code),
            can be 503 code, when Splunk is restarting and KV Store is not
            yet initialized.
    """
    try:
        self._collection_data.delete_by_id(key)
    except binding.HTTPError as e:
        if e.status != 404:
            logging.error(f"Delete checkpoint failed: {traceback.format_exc()}.")
            raise

get(key)

Gets document with an id that equals to key.

Parameters:

Name Type Description Default
key str

id of the document to get.

 required

Raises:

Type Description
binding.HTTPError

When an error occurred in Splunk (not 404 code), can be 503 code, when Splunk is restarting and KV Store is not yet initialized.

Returns:

Type Description
Optional[Any]

Document data under key or None in case of no data.
Source code in solnlib/modular_input/checkpointer.py 
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
@utils.retry(exceptions=[binding.HTTPError])
def get(self, key: str) -> Optional[Any]:
    """Gets document with an id that equals to `key`.

    Arguments:
        key: `id` of the document to get.

    Raises:
        binding.HTTPError: When an error occurred in Splunk (not 404 code),
            can be 503 code, when Splunk is restarting and KV Store is not
            yet initialized.

    Returns:
        Document data under `key` or `None` in case of no data.
    """
    try:
        record = self._collection_data.query_by_id(key)
    except binding.HTTPError as e:
        if e.status != 404:
            logging.error(f"Get checkpoint failed: {traceback.format_exc()}.")
            raise
        return None
    return json.loads(record["state"])

update(key, state)

Updates document with an id that equals to key and state as document data.

Parameters:

Name Type Description Default
key str

id of the document to update.

 required
state Any

Document data to update. It can be integer, string, or a dict, or anything that can be an argument to json.dumps.

 required

Raises:

Type Description
binding.HTTPError

when an error occurred in Splunk, for example, when Splunk is restarting and KV Store is not yet initialized.
Source code in solnlib/modular_input/checkpointer.py 
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
@utils.retry(exceptions=[binding.HTTPError])
def update(self, key: str, state: Any) -> None:
    """Updates document with an id that equals to `key` and `state` as
    document data.

    Arguments:
        key: `id` of the document to update.
        state: Document data to update. It can be integer, string,
            or a dict, or anything that can be an argument to `json.dumps`.

    Raises:
        binding.HTTPError: when an error occurred in Splunk, for example,
            when Splunk is restarting and KV Store is not yet initialized.
    """
    record = {"_key": key, "state": json.dumps(state)}
    self._collection_data.batch_save(record)

event.py

This module provides Splunk modular input event encapsulation.

__all__ = ['EventException', 'XMLEvent', 'HECEvent'] module-attribute

Event

Base class of modular input event.

Source code in solnlib/modular_input/event.py 
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
class Event:
    """Base class of modular input event."""

    def __init__(
        self,
        data: dict,
        time: float = None,
        index: str = None,
        host: str = None,
        source: str = None,
        sourcetype: str = None,
        fields: dict = None,
        stanza: str = None,
        unbroken: bool = False,
        done: bool = False,
    ):
        """Modular input event.

        Arguments:
            data: Event data.
            time: (optional) Event timestamp, default is None.
            index: (optional) The index event will be written to, default is None.
            host: (optional) Event host, default is None.
            source: (optional) Event source, default is None.
            sourcetype: (optional) Event sourcetype, default is None.
            fields: (optional) Event fields, default is None.
            stanza: (optional) Event stanza name, default is None.
            unbroken: (optional) Event unbroken flag, default is False.
            done: (optional) The last unbroken event, default is False.

        Examples:
           >>> event = Event(
           >>>     data='This is a test data.',
           >>>     time=1372274622.493,
           >>>     index='main',
           >>>     host='localhost',
           >>>     source='Splunk',
           >>>     sourcetype='misc',
           >>>     fields= {'Cloud':'AWS','region': 'us-west-1'},
           >>>     stanza='test_scheme://test',
           >>>     unbroken=True,
           >>>     done=True)
        """

        self._data = data
        self._time = "%.3f" % time if time else None
        self._index = index
        self._host = host
        self._source = source
        self._sourcetype = sourcetype
        if fields:
            self._fields = fields
        self._stanza = stanza
        if not unbroken and done:
            raise EventException('Invalid combination of "unbroken" and "done".')
        self._unbroken = unbroken
        self._done = done

    def __str__(self):
        event = {
            "data": self._data,
            "time": float(self._time) if self._time else self._time,
            "index": self._index,
            "host": self._host,
            "source": self._source,
            "sourcetype": self._sourcetype,
            "stanza": self._stanza,
            "unbroken": self._unbroken,
            "done": self._done,
        }

        if hasattr(self, "_fields"):
            event["fields"] = self._fields

        return json.dumps(event)

    @classmethod
    def format_events(cls, events: List) -> List:
        """Format events to list of string.

        Arguments:
            events: List of events to format.

        Returns:
            List of formatted events string.
        """

        raise EventException('Unimplemented "format_events".')

__init__(data, time=None, index=None, host=None, source=None, sourcetype=None, fields=None, stanza=None, unbroken=False, done=False)

Modular input event.

Parameters:

Name Type Description Default
data dict

Event data.

 required
time float

(optional) Event timestamp, default is None.

 None
index str

(optional) The index event will be written to, default is None.

 None
host str

(optional) Event host, default is None.

 None
source str

(optional) Event source, default is None.

 None
sourcetype str

(optional) Event sourcetype, default is None.

 None
fields dict

(optional) Event fields, default is None.

 None
stanza str

(optional) Event stanza name, default is None.

 None
unbroken bool

(optional) Event unbroken flag, default is False.

 False
done bool

(optional) The last unbroken event, default is False.

 False

Examples:

>>> event = Event(
>>>     data='This is a test data.',
>>>     time=1372274622.493,
>>>     index='main',
>>>     host='localhost',
>>>     source='Splunk',
>>>     sourcetype='misc',
>>>     fields= {'Cloud':'AWS','region': 'us-west-1'},
>>>     stanza='test_scheme://test',
>>>     unbroken=True,
>>>     done=True)
Source code in solnlib/modular_input/event.py 
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
def __init__(
    self,
    data: dict,
    time: float = None,
    index: str = None,
    host: str = None,
    source: str = None,
    sourcetype: str = None,
    fields: dict = None,
    stanza: str = None,
    unbroken: bool = False,
    done: bool = False,
):
    """Modular input event.

    Arguments:
        data: Event data.
        time: (optional) Event timestamp, default is None.
        index: (optional) The index event will be written to, default is None.
        host: (optional) Event host, default is None.
        source: (optional) Event source, default is None.
        sourcetype: (optional) Event sourcetype, default is None.
        fields: (optional) Event fields, default is None.
        stanza: (optional) Event stanza name, default is None.
        unbroken: (optional) Event unbroken flag, default is False.
        done: (optional) The last unbroken event, default is False.

    Examples:
       >>> event = Event(
       >>>     data='This is a test data.',
       >>>     time=1372274622.493,
       >>>     index='main',
       >>>     host='localhost',
       >>>     source='Splunk',
       >>>     sourcetype='misc',
       >>>     fields= {'Cloud':'AWS','region': 'us-west-1'},
       >>>     stanza='test_scheme://test',
       >>>     unbroken=True,
       >>>     done=True)
    """

    self._data = data
    self._time = "%.3f" % time if time else None
    self._index = index
    self._host = host
    self._source = source
    self._sourcetype = sourcetype
    if fields:
        self._fields = fields
    self._stanza = stanza
    if not unbroken and done:
        raise EventException('Invalid combination of "unbroken" and "done".')
    self._unbroken = unbroken
    self._done = done

__str__()

Source code in solnlib/modular_input/event.py 
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
def __str__(self):
    event = {
        "data": self._data,
        "time": float(self._time) if self._time else self._time,
        "index": self._index,
        "host": self._host,
        "source": self._source,
        "sourcetype": self._sourcetype,
        "stanza": self._stanza,
        "unbroken": self._unbroken,
        "done": self._done,
    }

    if hasattr(self, "_fields"):
        event["fields"] = self._fields

    return json.dumps(event)

format_events(events) classmethod

Format events to list of string.

Parameters:

Name Type Description Default
events List

List of events to format.

 required

Returns:

Type Description
List

List of formatted events string.
Source code in solnlib/modular_input/event.py 
108
109
110
111
112
113
114
115
116
117
118
119
@classmethod
def format_events(cls, events: List) -> List:
    """Format events to list of string.

    Arguments:
        events: List of events to format.

    Returns:
        List of formatted events string.
    """

    raise EventException('Unimplemented "format_events".')

EventException

Bases: Exception

Source code in solnlib/modular_input/event.py 
28
29
class EventException(Exception):
    pass

HECEvent

Bases: Event

HEC event.

Source code in solnlib/modular_input/event.py 
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
class HECEvent(Event):
    """HEC event."""

    max_hec_event_length = 1000000

    def _to_hec(self, event_field):
        event = {}
        event[event_field] = self._data
        if self._time:
            event["time"] = float(self._time)
        if self._index:
            event["index"] = self._index
        if self._host:
            event["host"] = self._host
        if self._source:
            event["source"] = self._source
        if self._sourcetype:
            event["sourcetype"] = self._sourcetype
        if hasattr(self, "_fields"):
            event["fields"] = self._fields

        return json.dumps(event, ensure_ascii=False)

    @classmethod
    def format_events(cls, events: List, event_field: str = "event") -> List:
        """Format events to list of string.

        Arguments:
            events: List of events to format.
            event_field: Event field.

        Returns:
            List of formatted events string, example::

                [
                    '{"index": "main", ... "event": {"kk": [1, 2, 3]}}\\n
                    {"index": "main", ... "event": {"kk": [3, 2, 3]}}',
                '...'
                ]
        """

        size = 0
        new_events, batched_events = [], []
        events = [event._to_hec(event_field) for event in events]
        for event in events:
            new_length = size + len(event) + len(batched_events) - 1
            if new_length >= cls.max_hec_event_length:
                if batched_events:
                    new_events.append("\n".join(batched_events))
                del batched_events[:]
                size = 0

            batched_events.append(event)
            size = size + len(event)
        if batched_events:
            new_events.append("\n".join(batched_events))

        return new_events

max_hec_event_length = 1000000 instance-attribute class-attribute

format_events(events, event_field='event') classmethod

Format events to list of string.

Parameters:

Name Type Description Default
events List

List of events to format.

 required
event_field str

Event field.

 'event'

Returns:

Type Description
List

List of formatted events string, example::

[ ‘{“index”: “main”, … “event”: {“kk”: [1, 2, 3]}}\n {“index”: “main”, … “event”: {“kk”: [3, 2, 3]}}’, ‘…’ ]
Source code in solnlib/modular_input/event.py 
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
@classmethod
def format_events(cls, events: List, event_field: str = "event") -> List:
    """Format events to list of string.

    Arguments:
        events: List of events to format.
        event_field: Event field.

    Returns:
        List of formatted events string, example::

            [
                '{"index": "main", ... "event": {"kk": [1, 2, 3]}}\\n
                {"index": "main", ... "event": {"kk": [3, 2, 3]}}',
            '...'
            ]
    """

    size = 0
    new_events, batched_events = [], []
    events = [event._to_hec(event_field) for event in events]
    for event in events:
        new_length = size + len(event) + len(batched_events) - 1
        if new_length >= cls.max_hec_event_length:
            if batched_events:
                new_events.append("\n".join(batched_events))
            del batched_events[:]
            size = 0

        batched_events.append(event)
        size = size + len(event)
    if batched_events:
        new_events.append("\n".join(batched_events))

    return new_events

XMLEvent

Bases: Event

XML event.

Source code in solnlib/modular_input/event.py 
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
class XMLEvent(Event):
    """XML event."""

    def _to_xml(self):
        _event = ET.Element("event")
        if self._stanza:
            _event.set("stanza", self._stanza)
        if self._unbroken:
            _event.set("unbroken", str(int(self._unbroken)))

        if self._time:
            ET.SubElement(_event, "time").text = self._time

        sub_elements = [
            ("index", self._index),
            ("host", self._host),
            ("source", self._source),
            ("sourcetype", self._sourcetype),
        ]
        for node, value in sub_elements:
            if value:
                ET.SubElement(_event, node).text = value

        if isinstance(self._data, str):
            ET.SubElement(_event, "data").text = self._data
        else:
            ET.SubElement(_event, "data").text = json.dumps(self._data)

        if self._done:
            ET.SubElement(_event, "done")

        return _event

    @classmethod
    def format_events(cls, events: List) -> List:
        """Format events to list of string.

        Arguments:
            events: List of events to format.

        Returns:
            List of formatted events string, example::

                [
                    '<stream>
                    <event stanza="test_scheme://test" unbroken="1">
                    <time>1459919070.994</time>
                    <index>main</index>
                    <host>localhost</host>
                    <source>test</source>
                    <sourcetype>test</sourcetype>
                    <data>{"kk": [1, 2, 3]}</data>
                    <done />
                    </event>
                    <event stanza="test_scheme://test" unbroken="1">
                    <time>1459919082.961</time>
                    <index>main</index>
                    <host>localhost</host>
                    <source>test</source>
                    <sourcetype>test</sourcetype>
                    <data>{"kk": [3, 2, 3]}</data>
                    <done />
                    </event>
                    </stream>'
                ]
        """

        stream = ET.Element("stream")
        for event in events:
            stream.append(event._to_xml())

        return [
            defused_et.tostring(stream, encoding="utf-8", method="xml").decode("utf-8")
        ]

format_events(events) classmethod

Format events to list of string.

Parameters:

Name Type Description Default
events List

List of events to format.

 required

Returns:

Type Description
List

List of formatted events string, example::

[ ‘ main localhost test test {“kk”: [1, 2, 3]} main localhost test test {“kk”: [3, 2, 3]} ’ ]
Source code in solnlib/modular_input/event.py 
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
@classmethod
def format_events(cls, events: List) -> List:
    """Format events to list of string.

    Arguments:
        events: List of events to format.

    Returns:
        List of formatted events string, example::

            [
                '<stream>
                <event stanza="test_scheme://test" unbroken="1">
                <time>1459919070.994</time>
                <index>main</index>
                <host>localhost</host>
                <source>test</source>
                <sourcetype>test</sourcetype>
                <data>{"kk": [1, 2, 3]}</data>
                <done />
                </event>
                <event stanza="test_scheme://test" unbroken="1">
                <time>1459919082.961</time>
                <index>main</index>
                <host>localhost</host>
                <source>test</source>
                <sourcetype>test</sourcetype>
                <data>{"kk": [3, 2, 3]}</data>
                <done />
                </event>
                </stream>'
            ]
    """

    stream = ET.Element("stream")
    for event in events:
        stream.append(event._to_xml())

    return [
        defused_et.tostring(stream, encoding="utf-8", method="xml").decode("utf-8")
    ]

event_writer.py

This module provides two kinds of event writers (ClassicEventWriter, HECEventWriter) to write Splunk modular input events.

__all__ = ['ClassicEventWriter', 'HECEventWriter'] module-attribute

ClassicEventWriter

Bases: EventWriter

Classic event writer.

Use sys.stdout as the output.

Examples:

>>> from solnlib.modular_input import event_writer
>>> ew = event_writer.ClassicEventWriter()
>>> ew.write_events([event1, event2])
Source code in solnlib/modular_input/event_writer.py 
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
class ClassicEventWriter(EventWriter):
    """Classic event writer.

    Use sys.stdout as the output.

    Examples:
        >>> from solnlib.modular_input import event_writer
        >>> ew = event_writer.ClassicEventWriter()
        >>> ew.write_events([event1, event2])
    """

    description = "ClassicEventWriter"

    def __init__(self, lock: Union[threading.Lock, multiprocessing.Lock] = None):
        """Initializes ClassicEventWriter.

        Arguments:
            lock: (optional) lock to exclusively access stdout.
                by default, it is None and it will use threading safe lock.
                if user would like to make the lock multiple-process safe, user should
                pass in multiprocessing.Lock() instead
        """
        if lock is None:
            self._lock = threading.Lock()
        else:
            self._lock = lock

    def create_event(
        self,
        data: dict,
        time: float = None,
        index: str = None,
        host: str = None,
        source: str = None,
        sourcetype: str = None,
        fields: dict = None,
        stanza: str = None,
        unbroken: bool = False,
        done: bool = False,
    ):
        """Create a new XMLEvent object."""

        return XMLEvent(
            data,
            time=time,
            index=index,
            host=host,
            source=source,
            sourcetype=sourcetype,
            stanza=stanza,
            unbroken=unbroken,
            done=done,
        )

    def write_events(self, events):
        if not events:
            return

        stdout = sys.stdout

        data = "".join([event for event in XMLEvent.format_events(events)])
        with self._lock:
            stdout.write(data)
            stdout.flush()

description = 'ClassicEventWriter' instance-attribute class-attribute

__init__(lock=None)

Initializes ClassicEventWriter.

Parameters:

Name Type Description Default
lock Union[threading.Lock, multiprocessing.Lock]

(optional) lock to exclusively access stdout. by default, it is None and it will use threading safe lock. if user would like to make the lock multiple-process safe, user should pass in multiprocessing.Lock() instead

 None
Source code in solnlib/modular_input/event_writer.py 
123
124
125
126
127
128
129
130
131
132
133
134
135
def __init__(self, lock: Union[threading.Lock, multiprocessing.Lock] = None):
    """Initializes ClassicEventWriter.

    Arguments:
        lock: (optional) lock to exclusively access stdout.
            by default, it is None and it will use threading safe lock.
            if user would like to make the lock multiple-process safe, user should
            pass in multiprocessing.Lock() instead
    """
    if lock is None:
        self._lock = threading.Lock()
    else:
        self._lock = lock

create_event(data, time=None, index=None, host=None, source=None, sourcetype=None, fields=None, stanza=None, unbroken=False, done=False)

Create a new XMLEvent object.

Source code in solnlib/modular_input/event_writer.py 
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
def create_event(
    self,
    data: dict,
    time: float = None,
    index: str = None,
    host: str = None,
    source: str = None,
    sourcetype: str = None,
    fields: dict = None,
    stanza: str = None,
    unbroken: bool = False,
    done: bool = False,
):
    """Create a new XMLEvent object."""

    return XMLEvent(
        data,
        time=time,
        index=index,
        host=host,
        source=source,
        sourcetype=sourcetype,
        stanza=stanza,
        unbroken=unbroken,
        done=done,
    )

write_events(events)

Source code in solnlib/modular_input/event_writer.py 
164
165
166
167
168
169
170
171
172
173
def write_events(self, events):
    if not events:
        return

    stdout = sys.stdout

    data = "".join([event for event in XMLEvent.format_events(events)])
    with self._lock:
        stdout.write(data)
        stdout.flush()

EventWriter

Base class of event writer.

Source code in solnlib/modular_input/event_writer.py 
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
class EventWriter(metaclass=ABCMeta):
    """Base class of event writer."""

    description = "EventWriter"

    @abstractmethod
    def create_event(
        self,
        data: dict,
        time: float = None,
        index: str = None,
        host: str = None,
        source: str = None,
        sourcetype: str = None,
        fields: dict = None,
        stanza: str = None,
        unbroken: bool = False,
        done: bool = False,
    ) -> Union[XMLEvent, HECEvent]:
        """Create a new event.

        Arguments:
            data: Event data.
            time: (optional) Event timestamp, default is None.
            index: (optional) The index event will be written to, default is None.
            host: (optional) Event host, default is None.
            source: (optional) Event source, default is None.
            sourcetype: (optional) Event sourcetype, default is None.
            fields: (optional) Event fields, default is None.
            stanza: (optional) Event stanza name, default is None.
            unbroken: (optional) Event unbroken flag, default is False.
                It is only meaningful when for XMLEvent when using ClassicEventWriter.
            done: (optional) The last unbroken event, default is False.
                It is only meaningful when for XMLEvent when using ClassicEventWriter.

        Examples:
           >>> ew = event_writer.HECEventWriter(...)
           >>> event = ew.create_event(
           >>>     data='This is a test data.',
           >>>     time='%.3f' % 1372274622.493,
           >>>     index='main',
           >>>     host='localhost',
           >>>     source='Splunk',
           >>>     sourcetype='misc',
           >>>     fields={'accountid': '603514901691', 'Cloud': u'AWS'},
           >>>     stanza='test_scheme://test',
           >>>     unbroken=True,
           >>>     done=True)
        """

        pass

    @abstractmethod
    def write_events(self, events: List):
        """Write events.

        Arguments:
            events: List of events to write.

        Examples:
           >>> from solnlib.modular_input import event_writer
           >>> ew = event_writer.EventWriter(...)
           >>> ew.write_events([event1, event2])
        """

        pass

description = 'EventWriter' instance-attribute class-attribute

create_event(data, time=None, index=None, host=None, source=None, sourcetype=None, fields=None, stanza=None, unbroken=False, done=False) abstractmethod

Create a new event.

Parameters:

Name Type Description Default
data dict

Event data.

 required
time float

(optional) Event timestamp, default is None.

 None
index str

(optional) The index event will be written to, default is None.

 None
host str

(optional) Event host, default is None.

 None
source str

(optional) Event source, default is None.

 None
sourcetype str

(optional) Event sourcetype, default is None.

 None
fields dict

(optional) Event fields, default is None.

 None
stanza str

(optional) Event stanza name, default is None.

 None
unbroken bool

(optional) Event unbroken flag, default is False. It is only meaningful when for XMLEvent when using ClassicEventWriter.

 False
done bool

(optional) The last unbroken event, default is False. It is only meaningful when for XMLEvent when using ClassicEventWriter.

 False

Examples:

>>> ew = event_writer.HECEventWriter(...)
>>> event = ew.create_event(
>>>     data='This is a test data.',
>>>     time='%.3f' % 1372274622.493,
>>>     index='main',
>>>     host='localhost',
>>>     source='Splunk',
>>>     sourcetype='misc',
>>>     fields={'accountid': '603514901691', 'Cloud': u'AWS'},
>>>     stanza='test_scheme://test',
>>>     unbroken=True,
>>>     done=True)
Source code in solnlib/modular_input/event_writer.py 
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
@abstractmethod
def create_event(
    self,
    data: dict,
    time: float = None,
    index: str = None,
    host: str = None,
    source: str = None,
    sourcetype: str = None,
    fields: dict = None,
    stanza: str = None,
    unbroken: bool = False,
    done: bool = False,
) -> Union[XMLEvent, HECEvent]:
    """Create a new event.

    Arguments:
        data: Event data.
        time: (optional) Event timestamp, default is None.
        index: (optional) The index event will be written to, default is None.
        host: (optional) Event host, default is None.
        source: (optional) Event source, default is None.
        sourcetype: (optional) Event sourcetype, default is None.
        fields: (optional) Event fields, default is None.
        stanza: (optional) Event stanza name, default is None.
        unbroken: (optional) Event unbroken flag, default is False.
            It is only meaningful when for XMLEvent when using ClassicEventWriter.
        done: (optional) The last unbroken event, default is False.
            It is only meaningful when for XMLEvent when using ClassicEventWriter.

    Examples:
       >>> ew = event_writer.HECEventWriter(...)
       >>> event = ew.create_event(
       >>>     data='This is a test data.',
       >>>     time='%.3f' % 1372274622.493,
       >>>     index='main',
       >>>     host='localhost',
       >>>     source='Splunk',
       >>>     sourcetype='misc',
       >>>     fields={'accountid': '603514901691', 'Cloud': u'AWS'},
       >>>     stanza='test_scheme://test',
       >>>     unbroken=True,
       >>>     done=True)
    """

    pass

write_events(events) abstractmethod

Write events.

Parameters:

Name Type Description Default
events List

List of events to write.

 required

Examples:

>>> from solnlib.modular_input import event_writer
>>> ew = event_writer.EventWriter(...)
>>> ew.write_events([event1, event2])
Source code in solnlib/modular_input/event_writer.py 
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
@abstractmethod
def write_events(self, events: List):
    """Write events.

    Arguments:
        events: List of events to write.

    Examples:
       >>> from solnlib.modular_input import event_writer
       >>> ew = event_writer.EventWriter(...)
       >>> ew.write_events([event1, event2])
    """

    pass

HECEventWriter

Bases: EventWriter

HEC event writer.

Use Splunk HEC as the output.

Examples:

>>> from solnlib.modular_input import event_writer
>>> ew = event_writer.HECEventWriter(hec_input_name, session_key)
>>> ew.write_events([event1, event2])
Source code in solnlib/modular_input/event_writer.py 
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
class HECEventWriter(EventWriter):
    """HEC event writer.

    Use Splunk HEC as the output.

    Examples:
        >>> from solnlib.modular_input import event_writer
        >>> ew = event_writer.HECEventWriter(hec_input_name, session_key)
        >>> ew.write_events([event1, event2])
    """

    WRITE_EVENT_RETRIES = 5
    HTTP_INPUT_CONFIG_ENDPOINT = "/servicesNS/nobody/splunk_httpinput/data/inputs/http"
    HTTP_EVENT_COLLECTOR_ENDPOINT = "/services/collector"
    TOO_MANY_REQUESTS = 429  # we exceeded rate limit
    SERVICE_UNAVAILABLE = 503  # remote service is temporary unavailable

    description = "HECEventWriter"

    headers = [("Content-Type", "application/json")]

    def __init__(
        self,
        hec_input_name: str,
        session_key: str,
        scheme: str = None,
        host: str = None,
        port: int = None,
        hec_uri: str = None,
        hec_token: str = None,
        global_settings_schema: bool = True,
        logger: logging.Logger = None,
        **context: dict
    ):
        """Initializes HECEventWriter.

        Arguments:
            hec_input_name: Splunk HEC input name.
            session_key: Splunk access token.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            hec_uri: (optional) If hec_uri and hec_token are provided, they will
                higher precedence than hec_input_name.
            hec_token: (optional) HEC token.
            global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
            logger: Logger object.
            context: Other configurations for Splunk rest client.
        """
        super().__init__()
        self._session_key = session_key
        if logger:
            self.logger = logger
        else:
            self.logger = logging

        if hec_uri and hec_token:
            scheme, host, hec_port = utils.extract_http_scheme_host_port(hec_uri)
        else:
            if not all([scheme, host, port]):
                scheme, host, port = get_splunkd_access_info()
            hec_port, hec_token = self._get_hec_config(
                hec_input_name, session_key, scheme, host, port, **context
            )

        if global_settings_schema:
            scheme = get_scheme_from_hec_settings()

        if not context.get("pool_connections"):
            context["pool_connections"] = 10

        if not context.get("pool_maxsize"):
            context["pool_maxsize"] = 10

        self._rest_client = rest_client.SplunkRestClient(
            hec_token, app="-", scheme=scheme, host=host, port=hec_port, **context
        )

    @staticmethod
    def create_from_token(
        hec_uri: str,
        hec_token: str,
        global_settings_schema: bool = False,
        **context: dict
    ) -> "HECEventWriter":
        """Given HEC URI and HEC token, create HECEventWriter object. This
        function simplifies the standalone mode HECEventWriter usage (not in a
        modinput).

        Arguments:
            hec_uri: HTTP Event Collector URI, like https://localhost:8088.
            hec_token: HTTP Event Collector token.
            global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
            context: Other configurations.

        Returns:
            Created HECEventWriter.
        """

        return HECEventWriter(
            None,
            None,
            None,
            None,
            None,
            hec_uri=hec_uri,
            hec_token=hec_token,
            global_settings_schema=global_settings_schema,
            **context
        )

    @staticmethod
    def create_from_input(
        hec_input_name: str,
        splunkd_uri: str,
        session_key: str,
        global_settings_schema: bool = False,
        **context: dict
    ) -> "HECEventWriter":
        """Given HEC input stanza name, splunkd URI and splunkd session key,
        create HECEventWriter object. HEC URI and token etc will be discovered
        from HEC input stanza. When hitting HEC event limit, the underlying
        code will increase the HEC event limit automatically by calling
        corresponding REST API against splunkd_uri by using session_key.

        Arguments:
            hec_input_name: Splunk HEC input name.
            splunkd_uri: Splunkd URI, like https://localhost:8089
            session_key: Splunkd access token.
            global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
            context: Other configurations.

        Returns:
            Created HECEventWriter.
        """

        scheme, host, port = utils.extract_http_scheme_host_port(splunkd_uri)
        return HECEventWriter(
            hec_input_name,
            session_key,
            scheme,
            host,
            port,
            global_settings_schema=global_settings_schema,
            **context
        )

    @staticmethod
    def create_from_token_with_session_key(
        splunkd_uri: str,
        session_key: str,
        hec_uri: str,
        hec_token: str,
        global_settings_schema: bool = False,
        **context: dict
    ) -> "HECEventWriter":
        """Given Splunkd URI, Splunkd session key, HEC URI and HEC token,
        create HECEventWriter object. When hitting HEC event limit, the event
        writer will increase the HEC event limit automatically by calling
        corresponding REST API against splunkd_uri by using session_key.

        Arguments:
            splunkd_uri: Splunkd URI, like https://localhost:8089.
            session_key: Splunkd access token.
            hec_uri: Http Event Collector URI, like https://localhost:8088.
            hec_token: Http Event Collector token.
            global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
            context: Other configurations.

        Returns:
            Created HECEventWriter.
        """

        scheme, host, port = utils.extract_http_scheme_host_port(splunkd_uri)
        return HECEventWriter(
            None,
            session_key,
            scheme,
            host,
            port,
            hec_uri=hec_uri,
            hec_token=hec_token,
            global_settings_schema=global_settings_schema,
            **context
        )

    @retry(exceptions=[binding.HTTPError])
    def _get_hec_config(
        self, hec_input_name, session_key, scheme, host, port, **context
    ):
        hc = HECConfig(session_key, scheme=scheme, host=host, port=port, **context)
        settings = hc.get_settings()
        if utils.is_true(settings.get("disabled")):
            # Enable HEC input
            self.logger.info("Enabling HEC")
            settings["disabled"] = "0"
            settings["enableSSL"] = context.get("hec_enablessl", "1")
            settings["port"] = context.get("hec_port", "8088")
            hc.update_settings(settings)

        hec_input = hc.get_input(hec_input_name)
        if not hec_input:
            # Create HEC input
            self.logger.info("Create HEC datainput, name=%s", hec_input_name)
            hinput = {
                "index": context.get("index", "main"),
            }

            if context.get("sourcetype"):
                hinput["sourcetype"] = context["sourcetype"]

            if context.get("token"):
                hinput["token"] = context["token"]

            if context.get("source"):
                hinput["source"] = context["source"]

            if context.get("host"):
                hinput["host"] = context["host"]

            hec_input = hc.create_input(hec_input_name, hinput)

        limits = hc.get_limits()
        HECEvent.max_hec_event_length = int(limits.get("max_content_length", 1000000))

        return settings["port"], hec_input["token"]

    def create_event(
        self,
        data: dict,
        time: float = None,
        index: str = None,
        host: str = None,
        source: str = None,
        sourcetype: str = None,
        fields: dict = None,
        stanza: str = None,
        unbroken: bool = False,
        done: bool = False,
    ) -> HECEvent:
        """Create a new HECEvent object.

        Arguments:
            data: Event data.
            time: (optional) Event timestamp, default is None.
            index: (optional) The index event will be written to, default is None.
            host: (optional) Event host, default is None.
            source: (optional) Event source, default is None.
            sourcetype: (optional) Event sourcetype, default is None.
            fields: (optional) Event fields, default is None.
            stanza: (optional) Event stanza name, default is None.
            unbroken: (optional) Event unbroken flag, default is False.
                It is only meaningful when for XMLEvent when using ClassicEventWriter.
            done: (optional) The last unbroken event, default is False.
                It is only meaningful when for XMLEvent when using ClassicEventWriter.

        Returns:
            Created HECEvent.
        """

        return HECEvent(
            data,
            time=time,
            index=index,
            host=host,
            source=source,
            sourcetype=sourcetype,
            fields=fields,
        )

    def write_events(
        self,
        events: List,
        retries: int = WRITE_EVENT_RETRIES,
        event_field: str = "event",
    ):
        """Write events to index in bulk.

        Arguments:
            events: List of events.
            retries: Number of retries for writing events to index.
            event_field: Event field.
        """
        if not events:
            return

        last_ex = None
        for event in HECEvent.format_events(events, event_field):
            for i in range(retries):
                try:
                    self._rest_client.post(
                        self.HTTP_EVENT_COLLECTOR_ENDPOINT,
                        body=event.encode("utf-8"),
                        headers=self.headers,
                    )
                except binding.HTTPError as e:
                    self.logger.warn(
                        "Write events through HEC failed. Status=%s", e.status
                    )
                    last_ex = e
                    if e.status in [self.TOO_MANY_REQUESTS, self.SERVICE_UNAVAILABLE]:
                        # wait time for n retries: 10, 20, 40, 80, 80, 80, 80, ....
                        sleep_time = min(((2 ** (i + 1)) * 5), 80)
                        if i < retries - 1:
                            random_millisecond = randint(0, 1000) / 1000.0
                            time.sleep(sleep_time + random_millisecond)
                    else:
                        raise last_ex
                else:
                    break
            else:
                # When failed after retry, we reraise the exception
                # to exit the function to let client handle this situation
                self.logger.error(
                    "Write events through HEC failed: %s. status=%s",
                    traceback.format_exc(),
                    last_ex.status,
                )
                raise last_ex

HTTP_EVENT_COLLECTOR_ENDPOINT = '/services/collector' instance-attribute class-attribute

HTTP_INPUT_CONFIG_ENDPOINT = '/servicesNS/nobody/splunk_httpinput/data/inputs/http' instance-attribute class-attribute

SERVICE_UNAVAILABLE = 503 instance-attribute class-attribute

TOO_MANY_REQUESTS = 429 instance-attribute class-attribute

WRITE_EVENT_RETRIES = 5 instance-attribute class-attribute

description = 'HECEventWriter' instance-attribute class-attribute

headers = [('Content-Type', 'application/json')] instance-attribute class-attribute

logger = logger instance-attribute

__init__(hec_input_name, session_key, scheme=None, host=None, port=None, hec_uri=None, hec_token=None, global_settings_schema=True, logger=None, **context)

Initializes HECEventWriter.

Parameters:

Name Type Description Default
hec_input_name str

Splunk HEC input name.

 required
session_key str

Splunk access token.

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
hec_uri str

(optional) If hec_uri and hec_token are provided, they will higher precedence than hec_input_name.

 None
hec_token str

(optional) HEC token.

 None
global_settings_schema bool

(optional) if True, scheme will be set based on HEC global settings, default False.

 True
logger logging.Logger

Logger object.

 None
context dict

Other configurations for Splunk rest client.

 {}
Source code in solnlib/modular_input/event_writer.py 
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
def __init__(
    self,
    hec_input_name: str,
    session_key: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    hec_uri: str = None,
    hec_token: str = None,
    global_settings_schema: bool = True,
    logger: logging.Logger = None,
    **context: dict
):
    """Initializes HECEventWriter.

    Arguments:
        hec_input_name: Splunk HEC input name.
        session_key: Splunk access token.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        hec_uri: (optional) If hec_uri and hec_token are provided, they will
            higher precedence than hec_input_name.
        hec_token: (optional) HEC token.
        global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
        logger: Logger object.
        context: Other configurations for Splunk rest client.
    """
    super().__init__()
    self._session_key = session_key
    if logger:
        self.logger = logger
    else:
        self.logger = logging

    if hec_uri and hec_token:
        scheme, host, hec_port = utils.extract_http_scheme_host_port(hec_uri)
    else:
        if not all([scheme, host, port]):
            scheme, host, port = get_splunkd_access_info()
        hec_port, hec_token = self._get_hec_config(
            hec_input_name, session_key, scheme, host, port, **context
        )

    if global_settings_schema:
        scheme = get_scheme_from_hec_settings()

    if not context.get("pool_connections"):
        context["pool_connections"] = 10

    if not context.get("pool_maxsize"):
        context["pool_maxsize"] = 10

    self._rest_client = rest_client.SplunkRestClient(
        hec_token, app="-", scheme=scheme, host=host, port=hec_port, **context
    )

create_event(data, time=None, index=None, host=None, source=None, sourcetype=None, fields=None, stanza=None, unbroken=False, done=False)

Create a new HECEvent object.

Parameters:

Name Type Description Default
data dict

Event data.

 required
time float

(optional) Event timestamp, default is None.

 None
index str

(optional) The index event will be written to, default is None.

 None
host str

(optional) Event host, default is None.

 None
source str

(optional) Event source, default is None.

 None
sourcetype str

(optional) Event sourcetype, default is None.

 None
fields dict

(optional) Event fields, default is None.

 None
stanza str

(optional) Event stanza name, default is None.

 None
unbroken bool

(optional) Event unbroken flag, default is False. It is only meaningful when for XMLEvent when using ClassicEventWriter.

 False
done bool

(optional) The last unbroken event, default is False. It is only meaningful when for XMLEvent when using ClassicEventWriter.

 False

Returns:

Type Description
HECEvent

Created HECEvent.
Source code in solnlib/modular_input/event_writer.py 
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
def create_event(
    self,
    data: dict,
    time: float = None,
    index: str = None,
    host: str = None,
    source: str = None,
    sourcetype: str = None,
    fields: dict = None,
    stanza: str = None,
    unbroken: bool = False,
    done: bool = False,
) -> HECEvent:
    """Create a new HECEvent object.

    Arguments:
        data: Event data.
        time: (optional) Event timestamp, default is None.
        index: (optional) The index event will be written to, default is None.
        host: (optional) Event host, default is None.
        source: (optional) Event source, default is None.
        sourcetype: (optional) Event sourcetype, default is None.
        fields: (optional) Event fields, default is None.
        stanza: (optional) Event stanza name, default is None.
        unbroken: (optional) Event unbroken flag, default is False.
            It is only meaningful when for XMLEvent when using ClassicEventWriter.
        done: (optional) The last unbroken event, default is False.
            It is only meaningful when for XMLEvent when using ClassicEventWriter.

    Returns:
        Created HECEvent.
    """

    return HECEvent(
        data,
        time=time,
        index=index,
        host=host,
        source=source,
        sourcetype=sourcetype,
        fields=fields,
    )

create_from_input(hec_input_name, splunkd_uri, session_key, global_settings_schema=False, **context) staticmethod

Given HEC input stanza name, splunkd URI and splunkd session key, create HECEventWriter object. HEC URI and token etc will be discovered from HEC input stanza. When hitting HEC event limit, the underlying code will increase the HEC event limit automatically by calling corresponding REST API against splunkd_uri by using session_key.

Parameters:

Name Type Description Default
hec_input_name str

Splunk HEC input name.

 required
splunkd_uri str

Splunkd URI, like https://localhost:8089

 required
session_key str

Splunkd access token.

 required
global_settings_schema bool

(optional) if True, scheme will be set based on HEC global settings, default False.

 False
context dict

Other configurations.

 {}

Returns:

Type Description
HECEventWriter

Created HECEventWriter.
Source code in solnlib/modular_input/event_writer.py 
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
@staticmethod
def create_from_input(
    hec_input_name: str,
    splunkd_uri: str,
    session_key: str,
    global_settings_schema: bool = False,
    **context: dict
) -> "HECEventWriter":
    """Given HEC input stanza name, splunkd URI and splunkd session key,
    create HECEventWriter object. HEC URI and token etc will be discovered
    from HEC input stanza. When hitting HEC event limit, the underlying
    code will increase the HEC event limit automatically by calling
    corresponding REST API against splunkd_uri by using session_key.

    Arguments:
        hec_input_name: Splunk HEC input name.
        splunkd_uri: Splunkd URI, like https://localhost:8089
        session_key: Splunkd access token.
        global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
        context: Other configurations.

    Returns:
        Created HECEventWriter.
    """

    scheme, host, port = utils.extract_http_scheme_host_port(splunkd_uri)
    return HECEventWriter(
        hec_input_name,
        session_key,
        scheme,
        host,
        port,
        global_settings_schema=global_settings_schema,
        **context
    )

create_from_token(hec_uri, hec_token, global_settings_schema=False, **context) staticmethod

Given HEC URI and HEC token, create HECEventWriter object. This function simplifies the standalone mode HECEventWriter usage (not in a modinput).

Parameters:

Name Type Description Default
hec_uri str

HTTP Event Collector URI, like https://localhost:8088.

 required
hec_token str

HTTP Event Collector token.

 required
global_settings_schema bool

(optional) if True, scheme will be set based on HEC global settings, default False.

 False
context dict

Other configurations.

 {}

Returns:

Type Description
HECEventWriter

Created HECEventWriter.
Source code in solnlib/modular_input/event_writer.py 
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
@staticmethod
def create_from_token(
    hec_uri: str,
    hec_token: str,
    global_settings_schema: bool = False,
    **context: dict
) -> "HECEventWriter":
    """Given HEC URI and HEC token, create HECEventWriter object. This
    function simplifies the standalone mode HECEventWriter usage (not in a
    modinput).

    Arguments:
        hec_uri: HTTP Event Collector URI, like https://localhost:8088.
        hec_token: HTTP Event Collector token.
        global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
        context: Other configurations.

    Returns:
        Created HECEventWriter.
    """

    return HECEventWriter(
        None,
        None,
        None,
        None,
        None,
        hec_uri=hec_uri,
        hec_token=hec_token,
        global_settings_schema=global_settings_schema,
        **context
    )

create_from_token_with_session_key(splunkd_uri, session_key, hec_uri, hec_token, global_settings_schema=False, **context) staticmethod

Given Splunkd URI, Splunkd session key, HEC URI and HEC token, create HECEventWriter object. When hitting HEC event limit, the event writer will increase the HEC event limit automatically by calling corresponding REST API against splunkd_uri by using session_key.

Parameters:

Name Type Description Default
splunkd_uri str

Splunkd URI, like https://localhost:8089.

 required
session_key str

Splunkd access token.

 required
hec_uri str

Http Event Collector URI, like https://localhost:8088.

 required
hec_token str

Http Event Collector token.

 required
global_settings_schema bool

(optional) if True, scheme will be set based on HEC global settings, default False.

 False
context dict

Other configurations.

 {}

Returns:

Type Description
HECEventWriter

Created HECEventWriter.
Source code in solnlib/modular_input/event_writer.py 
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
@staticmethod
def create_from_token_with_session_key(
    splunkd_uri: str,
    session_key: str,
    hec_uri: str,
    hec_token: str,
    global_settings_schema: bool = False,
    **context: dict
) -> "HECEventWriter":
    """Given Splunkd URI, Splunkd session key, HEC URI and HEC token,
    create HECEventWriter object. When hitting HEC event limit, the event
    writer will increase the HEC event limit automatically by calling
    corresponding REST API against splunkd_uri by using session_key.

    Arguments:
        splunkd_uri: Splunkd URI, like https://localhost:8089.
        session_key: Splunkd access token.
        hec_uri: Http Event Collector URI, like https://localhost:8088.
        hec_token: Http Event Collector token.
        global_settings_schema: (optional) if True, scheme will be set based on HEC global settings, default False.
        context: Other configurations.

    Returns:
        Created HECEventWriter.
    """

    scheme, host, port = utils.extract_http_scheme_host_port(splunkd_uri)
    return HECEventWriter(
        None,
        session_key,
        scheme,
        host,
        port,
        hec_uri=hec_uri,
        hec_token=hec_token,
        global_settings_schema=global_settings_schema,
        **context
    )

write_events(events, retries=WRITE_EVENT_RETRIES, event_field='event')

Write events to index in bulk.

Parameters:

Name Type Description Default
events List

List of events.

 required
retries int

Number of retries for writing events to index.

 WRITE_EVENT_RETRIES
event_field str

Event field.

 'event'
Source code in solnlib/modular_input/event_writer.py 
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
def write_events(
    self,
    events: List,
    retries: int = WRITE_EVENT_RETRIES,
    event_field: str = "event",
):
    """Write events to index in bulk.

    Arguments:
        events: List of events.
        retries: Number of retries for writing events to index.
        event_field: Event field.
    """
    if not events:
        return

    last_ex = None
    for event in HECEvent.format_events(events, event_field):
        for i in range(retries):
            try:
                self._rest_client.post(
                    self.HTTP_EVENT_COLLECTOR_ENDPOINT,
                    body=event.encode("utf-8"),
                    headers=self.headers,
                )
            except binding.HTTPError as e:
                self.logger.warn(
                    "Write events through HEC failed. Status=%s", e.status
                )
                last_ex = e
                if e.status in [self.TOO_MANY_REQUESTS, self.SERVICE_UNAVAILABLE]:
                    # wait time for n retries: 10, 20, 40, 80, 80, 80, 80, ....
                    sleep_time = min(((2 ** (i + 1)) * 5), 80)
                    if i < retries - 1:
                        random_millisecond = randint(0, 1000) / 1000.0
                        time.sleep(sleep_time + random_millisecond)
                else:
                    raise last_ex
            else:
                break
        else:
            # When failed after retry, we reraise the exception
            # to exit the function to let client handle this situation
            self.logger.error(
                "Write events through HEC failed: %s. status=%s",
                traceback.format_exc(),
                last_ex.status,
            )
            raise last_ex

modular_input.py

This module provides a base class of Splunk modular input.

__all__ = ['ModularInputException', 'ModularInput'] module-attribute

ModularInput

Base class of Splunk modular input.

It’s a base modular input, it should be inherited by sub modular input. For sub modular input, properties: ‘app’, ‘name’, ‘title’ and ‘description’ must be overriden, also there are some other optional properties can be overriden like: ‘use_external_validation’, ‘use_single_instance’, ‘use_kvstore_checkpointer’ and ‘use_hec_event_writer’.

Notes: If you set ‘KVStoreCheckpointer’ or ‘use_hec_event_writer’ to True, you must override the corresponding ‘kvstore_checkpointer_collection_name’ and ‘hec_input_name’.

Examples:

>>> class TestModularInput(ModularInput):
>>>     app = 'TestApp'
>>>     name = 'test_modular_input'
>>>     title = 'Test modular input'
>>>     description = 'This is a test modular input'
>>>     use_external_validation = True
>>>     use_single_instance = False
>>>     use_kvstore_checkpointer = True
>>>     kvstore_checkpointer_collection_name = 'TestCheckpoint'
>>>     use_hec_event_writer = True
>>>     hec_input_name = 'TestEventWriter'
>>>
>>>     def extra_arguments(self):
>>>         ... .. .
>>>
>>>     def do_validation(self, parameters):
>>>         ... .. .
>>>
>>>     def do_run(self, inputs):
>>>         ... .. .
>>>
>>> if __name__ == '__main__':
>>>     md = TestModularInput()
>>>     md.execute()
Source code in solnlib/modular_input/modular_input.py 
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
class ModularInput(metaclass=ABCMeta):
    """Base class of Splunk modular input.

    It's a base modular input, it should be inherited by sub modular input. For
    sub modular input, properties: 'app', 'name', 'title' and 'description' must
    be overriden, also there are some other optional properties can be overriden
    like: 'use_external_validation', 'use_single_instance', 'use_kvstore_checkpointer'
    and 'use_hec_event_writer'.

    Notes: If you set 'KVStoreCheckpointer' or 'use_hec_event_writer' to True,
    you must override the corresponding 'kvstore_checkpointer_collection_name'
    and 'hec_input_name'.

    Examples:

       >>> class TestModularInput(ModularInput):
       >>>     app = 'TestApp'
       >>>     name = 'test_modular_input'
       >>>     title = 'Test modular input'
       >>>     description = 'This is a test modular input'
       >>>     use_external_validation = True
       >>>     use_single_instance = False
       >>>     use_kvstore_checkpointer = True
       >>>     kvstore_checkpointer_collection_name = 'TestCheckpoint'
       >>>     use_hec_event_writer = True
       >>>     hec_input_name = 'TestEventWriter'
       >>>
       >>>     def extra_arguments(self):
       >>>         ... .. .
       >>>
       >>>     def do_validation(self, parameters):
       >>>         ... .. .
       >>>
       >>>     def do_run(self, inputs):
       >>>         ... .. .
       >>>
       >>> if __name__ == '__main__':
       >>>     md = TestModularInput()
       >>>     md.execute()
    """

    # App name, must be overridden
    app = None
    # Modular input name, must be overridden
    name = None
    # Modular input scheme title, must be overridden
    title = None
    # Modular input scheme description, must be overridden
    description = None
    # Modular input scheme use external validation, default is False
    use_external_validation = False
    # Modular input scheme use single instance mode, default is False
    use_single_instance = False
    # Use kvstore as checkpointer, default is True
    use_kvstore_checkpointer = True
    # Collection name of kvstore checkpointer, must be overridden if
    # use_kvstore_checkpointer is True
    kvstore_checkpointer_collection_name = None
    # Use hec event writer
    use_hec_event_writer = True
    # Input name of Splunk HEC, must be overridden if use_hec_event_writer
    # is True
    hec_input_name = None
    hec_global_settings_schema = False

    def __init__(self):
        # Validate properties
        self._validate_properties()
        # Modular input state
        self.should_exit = False
        # Metadata
        self.server_host_name = None
        self.server_uri = None
        self.server_scheme = None
        self.server_host = None
        self.server_port = None
        self.session_key = None
        # Modular input config name
        self.config_name = None
        # Checkpoint dir
        self._checkpoint_dir = None
        # Checkpointer
        self._checkpointer = None
        # Orphan process monitor
        self._orphan_monitor = None
        # Event writer
        self._event_writer = None

    def _validate_properties(self):
        if not all([self.app, self.name, self.title, self.description]):
            raise ModularInputException(
                'Attributes: "app", "name", "title", "description" must '
                "be overriden."
            )

        if self.use_kvstore_checkpointer:
            if self.kvstore_checkpointer_collection_name is None:
                raise ModularInputException(
                    'Attribute: "kvstore_checkpointer_collection_name" must'
                    'be overriden if "use_kvstore_checkpointer" is True".'
                )
            elif self.kvstore_checkpointer_collection_name.strip() == "":
                raise ModularInputException(
                    'Attribute: "kvstore_checkpointer_collection_name" can'
                    " not be empty."
                )

        if self.use_hec_event_writer:
            if self.hec_input_name is None:
                raise ModularInputException(
                    'Attribute: "hec_input_name" must be overriden '
                    'if "use_hec_event_writer" is True.'
                )
            elif self.hec_input_name.strip() == "":
                raise ModularInputException(
                    'Attribute: "hec_input_name" can not be empty.'
                )

    @property
    def checkpointer(self) -> checkpointer.Checkpointer:
        """Get checkpointer object.

        The checkpointer returned depends on use_kvstore_checkpointer flag,
        if use_kvstore_checkpointer is true will return an KVStoreCheckpointer
        object else an FileCheckpointer object.

        Returns:
            A checkpointer object.
        """

        if self._checkpointer is not None:
            return self._checkpointer

        self._checkpointer = self._create_checkpointer()
        return self._checkpointer

    def _create_checkpointer(self):
        if self.use_kvstore_checkpointer:
            checkpointer_name = ":".join(
                [self.app, self.config_name, self.kvstore_checkpointer_collection_name]
            )
            try:
                return checkpointer.KVStoreCheckpointer(
                    checkpointer_name,
                    self.session_key,
                    self.app,
                    owner="nobody",
                    scheme=self.server_scheme,
                    host=self.server_host,
                    port=self.server_port,
                )
            except binding.HTTPError:
                logging.error(
                    "Failed to init kvstore checkpointer: %s.", traceback.format_exc()
                )
                raise
        else:
            return checkpointer.FileCheckpointer(self._checkpoint_dir)

    @property
    def event_writer(self) -> event_writer.EventWriter:
        """Get event writer object.

        The event writer returned depends on use_hec_event_writer flag,
        if use_hec_event_writer is true will return an HECEventWriter
        object else an ClassicEventWriter object.

        Returns:
            Event writer object.
        """

        if self._event_writer is not None:
            return self._event_writer

        self._event_writer = self._create_event_writer()
        return self._event_writer

    def _create_event_writer(self):
        if self.use_hec_event_writer:
            hec_input_name = ":".join([self.app, self.hec_input_name])
            try:
                return event_writer.HECEventWriter(
                    hec_input_name,
                    self.session_key,
                    scheme=self.server_scheme,
                    host=self.server_host,
                    port=self.server_port,
                    global_settings_schema=self.hec_global_settings_schema,
                )
            except binding.HTTPError:
                logging.error(
                    "Failed to init HECEventWriter: %s.", traceback.format_exc()
                )
                raise
        else:
            return event_writer.ClassicEventWriter()

    def _update_metadata(self, metadata):
        self.server_host_name = metadata["server_host"]
        splunkd = urlparse.urlsplit(metadata["server_uri"])
        self.server_uri = splunkd.geturl()
        self.server_scheme = splunkd.scheme
        self.server_host = splunkd.hostname
        self.server_port = splunkd.port
        self.session_key = metadata["session_key"]
        self._checkpoint_dir = metadata["checkpoint_dir"]

    def _do_scheme(self):
        scheme = Scheme(self.title)
        scheme.description = self.description
        scheme.use_external_validation = self.use_external_validation
        scheme.streaming_mode = Scheme.streaming_mode_xml
        scheme.use_single_instance = self.use_single_instance

        for argument in self.extra_arguments():
            name = argument["name"]
            title = argument.get("title", None)
            description = argument.get("description", None)
            validation = argument.get("validation", None)
            data_type = argument.get("data_type", Argument.data_type_string)
            required_on_edit = argument.get("required_on_edit", False)
            required_on_create = argument.get("required_on_create", False)

            scheme.add_argument(
                Argument(
                    name,
                    title=title,
                    description=description,
                    validation=validation,
                    data_type=data_type,
                    required_on_edit=required_on_edit,
                    required_on_create=required_on_create,
                )
            )

        return defused_et.tostring(scheme.to_xml(), encoding="unicode")

    def extra_arguments(self) -> List:
        """Extra arguments for modular input.

        Default implementation is returning an empty list.

        Returns:
            List of arguments like::

                [
                    {
                        'name': 'arg1',
                        'title': 'arg1 title',
                        'description': 'arg1 description',
                        'validation': 'arg1 validation statement',
                        'data_type': Argument.data_type_string,
                        'required_on_edit': False,
                        'required_on_create': False
                    },
                    {...},
                    {...}
                ]
        """

        return []

    def do_validation(self, parameters):
        """Handles external validation for modular input kinds.

        When Splunk calls a modular input script in validation mode, it will
        pass in an XML document giving information about the Splunk instance
        (so you can call back into it if needed) and the name and parameters
        of the proposed input. If this function does not throw an exception,
        the validation is assumed to succeed. Otherwise any errors thrown will
        be turned into a string and logged back to Splunk.

        Arguments:
            parameters: The parameters of input passed by splunkd.

        Raises:
            Exception: If validation is failed.
        """

        pass

    @abstractmethod
    def do_run(self, inputs: dict):
        """Runs this modular input.

        Arguments:
            inputs: Command line arguments passed to this modular input.
                For single instance mode, inputs like::

                    {
                    'stanza_name1': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                    'stanza_name2': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                    'stanza_name3': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                    }

                For multiple instance mode, inputs like::

                    {
                    'stanza_name1': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                    }
        """

        pass

    def register_teardown_handler(self, handler: Callable, *args):
        """Register teardown signal handler.

        Arguments:
            handler: Teardown signal handler.
            args: Arguments to the handler.

        Examples:
           >>> mi = ModularInput(...)
           >>> def teardown_handler(arg1, arg2, ...):
           >>>     ...
           >>> mi.register_teardown_handler(teardown_handler, arg1, arg2, ...)
        """

        def _teardown_handler(signum, frame):
            handler(*args)

        utils.handle_teardown_signals(_teardown_handler)

    def register_orphan_handler(self, handler: Callable, *args):
        """Register orphan process handler.

        Arguments:
            handler: Teardown signal handler.
            args: Arguments to the handler.

        Examples:
           >>> mi = ModularInput(...)
           >>> def orphan_handler(arg1, arg2, ...):
           >>>     ...
           >>> mi.register_orphan_handler(orphan_handler, arg1, arg2, ...)
        """

        def _orphan_handler():
            handler(*args)

        if self._orphan_monitor is None:
            self._orphan_monitor = OrphanProcessMonitor(_orphan_handler)
            self._orphan_monitor.start()

    def get_validation_definition(self) -> dict:
        """Get validation definition.

        This method can be overwritten to get validation definition from
        other input instead `stdin`.

        Returns:
            A dict object must contains `metadata` and `parameters`::

                example: {
                    'metadata': {
                    'session_key': 'iCKPS0cvmpyeJk...sdaf',
                    'server_host': 'test-test.com',
                    'server_uri': 'https://127.0.0.1:8089',
                    'checkpoint_dir': '/tmp'
                    },
                    parameters: {'args1': value1, 'args2': value2}
                }
        """

        validation_definition = ValidationDefinition.parse(sys.stdin)
        return {
            "metadata": validation_definition.metadata,
            "parameters": validation_definition.parameters,
        }

    def get_input_definition(self) -> dict:
        """Get input definition.

        This method can be overwritten to get input definition from
        other input instead `stdin`.

        Returns:
            A dict object must contain `metadata` and `inputs`::

                example: {
                    'metadata': {
                    'session_key': 'iCKPS0cvmpyeJk...sdaf',
                    'server_host': 'test-test.com',
                    'server_uri': 'https://127.0.0.1:8089',
                    'checkpoint_dir': '/tmp'
                    },
                    inputs: {
                    'stanza1': {'arg1': value1, 'arg2': value2},
                    'stanza2': {'arg1': value1, 'arg2': value2}
                    }
                }
        """

        input_definition = InputDefinition.parse(sys.stdin)
        return {
            "metadata": input_definition.metadata,
            "inputs": input_definition.inputs,
        }

    def execute(self):
        """Modular input entry.

        Examples:
           >>> class TestModularInput(ModularInput):
           >>>         ... .. .
           >>>
           >>> if __name__ == '__main__':
           >>>     md = TestModularInput()
           >>>     md.execute()
        """

        if len(sys.argv) == 1:
            try:
                input_definition = self.get_input_definition()
                self._update_metadata(input_definition["metadata"])
                if self.use_single_instance:
                    self.config_name = self.name
                else:
                    self.config_name = list(input_definition["inputs"].keys())[0]
                self.do_run(input_definition["inputs"])
                logging.info("Modular input: %s exit normally.", self.name)
                return 0
            except Exception:
                logging.error(
                    "Modular input: %s exit with exception: %s.",
                    self.name,
                    traceback.format_exc(),
                )
                return 1
            finally:
                # Stop orphan monitor if any
                if self._orphan_monitor:
                    self._orphan_monitor.stop()

        elif str(sys.argv[1]).lower() == "--scheme":
            sys.stdout.write(self._do_scheme())
            sys.stdout.flush()
            return 0

        elif sys.argv[1].lower() == "--validate-arguments":
            try:
                validation_definition = self.get_validation_definition()
                self._update_metadata(validation_definition["metadata"])
                self.do_validation(validation_definition["parameters"])
                return 0
            except Exception as e:
                logging.error(
                    "Modular input: %s validate arguments with exception: %s.",
                    self.name,
                    traceback.format_exc(),
                )
                root = ET.Element("error")
                ET.SubElement(root, "message").text = str(e)
                sys.stderr.write(defused_et.tostring(root))
                sys.stderr.flush()
                return 1
        else:
            logging.error(
                'Modular input: %s run with invalid arguments: "%s".',
                self.name,
                " ".join(sys.argv[1:]),
            )
            return 1

app = None instance-attribute class-attribute

checkpointer: checkpointer.Checkpointer property

Get checkpointer object.

The checkpointer returned depends on use_kvstore_checkpointer flag, if use_kvstore_checkpointer is true will return an KVStoreCheckpointer object else an FileCheckpointer object.

Returns:

Type Description
checkpointer.Checkpointer

A checkpointer object.

config_name = None instance-attribute

description = None instance-attribute class-attribute

event_writer: event_writer.EventWriter property

Get event writer object.

The event writer returned depends on use_hec_event_writer flag, if use_hec_event_writer is true will return an HECEventWriter object else an ClassicEventWriter object.

Returns:

Type Description
event_writer.EventWriter

Event writer object.

hec_global_settings_schema = False instance-attribute class-attribute

hec_input_name = None instance-attribute class-attribute

kvstore_checkpointer_collection_name = None instance-attribute class-attribute

name = None instance-attribute class-attribute

server_host = None instance-attribute

server_host_name = None instance-attribute

server_port = None instance-attribute

server_scheme = None instance-attribute

server_uri = None instance-attribute

session_key = None instance-attribute

should_exit = False instance-attribute

title = None instance-attribute class-attribute

use_external_validation = False instance-attribute class-attribute

use_hec_event_writer = True instance-attribute class-attribute

use_kvstore_checkpointer = True instance-attribute class-attribute

use_single_instance = False instance-attribute class-attribute

__init__()

Source code in solnlib/modular_input/modular_input.py 
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
def __init__(self):
    # Validate properties
    self._validate_properties()
    # Modular input state
    self.should_exit = False
    # Metadata
    self.server_host_name = None
    self.server_uri = None
    self.server_scheme = None
    self.server_host = None
    self.server_port = None
    self.session_key = None
    # Modular input config name
    self.config_name = None
    # Checkpoint dir
    self._checkpoint_dir = None
    # Checkpointer
    self._checkpointer = None
    # Orphan process monitor
    self._orphan_monitor = None
    # Event writer
    self._event_writer = None

do_run(inputs) abstractmethod

Runs this modular input.

Parameters:

Name Type Description Default
inputs dict

Command line arguments passed to this modular input. For single instance mode, inputs like::

{
'stanza_name1': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
'stanza_name2': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
'stanza_name3': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
}

For multiple instance mode, inputs like::

{
'stanza_name1': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
}
required
Source code in solnlib/modular_input/modular_input.py 
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
@abstractmethod
def do_run(self, inputs: dict):
    """Runs this modular input.

    Arguments:
        inputs: Command line arguments passed to this modular input.
            For single instance mode, inputs like::

                {
                'stanza_name1': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                'stanza_name2': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                'stanza_name3': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                }

            For multiple instance mode, inputs like::

                {
                'stanza_name1': {'arg1': 'arg1_value', 'arg2': 'arg2_value', ...}
                }
    """

    pass

do_validation(parameters)

Handles external validation for modular input kinds.

When Splunk calls a modular input script in validation mode, it will pass in an XML document giving information about the Splunk instance (so you can call back into it if needed) and the name and parameters of the proposed input. If this function does not throw an exception, the validation is assumed to succeed. Otherwise any errors thrown will be turned into a string and logged back to Splunk.

Parameters:

Name Type Description Default
parameters

The parameters of input passed by splunkd.

 required

Raises:

Type Description
Exception

If validation is failed.
Source code in solnlib/modular_input/modular_input.py 
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
def do_validation(self, parameters):
    """Handles external validation for modular input kinds.

    When Splunk calls a modular input script in validation mode, it will
    pass in an XML document giving information about the Splunk instance
    (so you can call back into it if needed) and the name and parameters
    of the proposed input. If this function does not throw an exception,
    the validation is assumed to succeed. Otherwise any errors thrown will
    be turned into a string and logged back to Splunk.

    Arguments:
        parameters: The parameters of input passed by splunkd.

    Raises:
        Exception: If validation is failed.
    """

    pass

execute()

Modular input entry.

Examples:

>>> class TestModularInput(ModularInput):
>>>         ... .. .
>>>
>>> if __name__ == '__main__':
>>>     md = TestModularInput()
>>>     md.execute()
Source code in solnlib/modular_input/modular_input.py 
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
def execute(self):
    """Modular input entry.

    Examples:
       >>> class TestModularInput(ModularInput):
       >>>         ... .. .
       >>>
       >>> if __name__ == '__main__':
       >>>     md = TestModularInput()
       >>>     md.execute()
    """

    if len(sys.argv) == 1:
        try:
            input_definition = self.get_input_definition()
            self._update_metadata(input_definition["metadata"])
            if self.use_single_instance:
                self.config_name = self.name
            else:
                self.config_name = list(input_definition["inputs"].keys())[0]
            self.do_run(input_definition["inputs"])
            logging.info("Modular input: %s exit normally.", self.name)
            return 0
        except Exception:
            logging.error(
                "Modular input: %s exit with exception: %s.",
                self.name,
                traceback.format_exc(),
            )
            return 1
        finally:
            # Stop orphan monitor if any
            if self._orphan_monitor:
                self._orphan_monitor.stop()

    elif str(sys.argv[1]).lower() == "--scheme":
        sys.stdout.write(self._do_scheme())
        sys.stdout.flush()
        return 0

    elif sys.argv[1].lower() == "--validate-arguments":
        try:
            validation_definition = self.get_validation_definition()
            self._update_metadata(validation_definition["metadata"])
            self.do_validation(validation_definition["parameters"])
            return 0
        except Exception as e:
            logging.error(
                "Modular input: %s validate arguments with exception: %s.",
                self.name,
                traceback.format_exc(),
            )
            root = ET.Element("error")
            ET.SubElement(root, "message").text = str(e)
            sys.stderr.write(defused_et.tostring(root))
            sys.stderr.flush()
            return 1
    else:
        logging.error(
            'Modular input: %s run with invalid arguments: "%s".',
            self.name,
            " ".join(sys.argv[1:]),
        )
        return 1

extra_arguments()

Extra arguments for modular input.

Default implementation is returning an empty list.

Returns:

Type Description
List

List of arguments like::

[ { ‘name’: ‘arg1’, ‘title’: ‘arg1 title’, ‘description’: ‘arg1 description’, ‘validation’: ‘arg1 validation statement’, ‘data_type’: Argument.data_type_string, ‘required_on_edit’: False, ‘required_on_create’: False }, {…}, {…} ]
Source code in solnlib/modular_input/modular_input.py 
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
def extra_arguments(self) -> List:
    """Extra arguments for modular input.

    Default implementation is returning an empty list.

    Returns:
        List of arguments like::

            [
                {
                    'name': 'arg1',
                    'title': 'arg1 title',
                    'description': 'arg1 description',
                    'validation': 'arg1 validation statement',
                    'data_type': Argument.data_type_string,
                    'required_on_edit': False,
                    'required_on_create': False
                },
                {...},
                {...}
            ]
    """

    return []

get_input_definition()

Get input definition.

This method can be overwritten to get input definition from other input instead stdin.

Returns:

Type Description
dict

A dict object must contain metadata and inputs::

example: { ‘metadata’: { ‘session_key’: ‘iCKPS0cvmpyeJk…sdaf’, ‘server_host’: ‘test-test.com’, ‘server_uri’: ‘https://127.0.0.1:8089’, ‘checkpoint_dir’: ‘/tmp’ }, inputs: { ‘stanza1’: {‘arg1’: value1, ‘arg2’: value2}, ‘stanza2’: {‘arg1’: value1, ‘arg2’: value2} } }
Source code in solnlib/modular_input/modular_input.py 
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
def get_input_definition(self) -> dict:
    """Get input definition.

    This method can be overwritten to get input definition from
    other input instead `stdin`.

    Returns:
        A dict object must contain `metadata` and `inputs`::

            example: {
                'metadata': {
                'session_key': 'iCKPS0cvmpyeJk...sdaf',
                'server_host': 'test-test.com',
                'server_uri': 'https://127.0.0.1:8089',
                'checkpoint_dir': '/tmp'
                },
                inputs: {
                'stanza1': {'arg1': value1, 'arg2': value2},
                'stanza2': {'arg1': value1, 'arg2': value2}
                }
            }
    """

    input_definition = InputDefinition.parse(sys.stdin)
    return {
        "metadata": input_definition.metadata,
        "inputs": input_definition.inputs,
    }

get_validation_definition()

Get validation definition.

This method can be overwritten to get validation definition from other input instead stdin.

Returns:

Type Description
dict

A dict object must contains metadata and parameters::

example: { ‘metadata’: { ‘session_key’: ‘iCKPS0cvmpyeJk…sdaf’, ‘server_host’: ‘test-test.com’, ‘server_uri’: ‘https://127.0.0.1:8089’, ‘checkpoint_dir’: ‘/tmp’ }, parameters: {‘args1’: value1, ‘args2’: value2} }
Source code in solnlib/modular_input/modular_input.py 
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
def get_validation_definition(self) -> dict:
    """Get validation definition.

    This method can be overwritten to get validation definition from
    other input instead `stdin`.

    Returns:
        A dict object must contains `metadata` and `parameters`::

            example: {
                'metadata': {
                'session_key': 'iCKPS0cvmpyeJk...sdaf',
                'server_host': 'test-test.com',
                'server_uri': 'https://127.0.0.1:8089',
                'checkpoint_dir': '/tmp'
                },
                parameters: {'args1': value1, 'args2': value2}
            }
    """

    validation_definition = ValidationDefinition.parse(sys.stdin)
    return {
        "metadata": validation_definition.metadata,
        "parameters": validation_definition.parameters,
    }

register_orphan_handler(handler, *args)

Register orphan process handler.

Parameters:

Name Type Description Default
handler Callable

Teardown signal handler.

 required
args

Arguments to the handler.

 ()

Examples:

>>> mi = ModularInput(...)
>>> def orphan_handler(arg1, arg2, ...):
>>>     ...
>>> mi.register_orphan_handler(orphan_handler, arg1, arg2, ...)
Source code in solnlib/modular_input/modular_input.py 
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
def register_orphan_handler(self, handler: Callable, *args):
    """Register orphan process handler.

    Arguments:
        handler: Teardown signal handler.
        args: Arguments to the handler.

    Examples:
       >>> mi = ModularInput(...)
       >>> def orphan_handler(arg1, arg2, ...):
       >>>     ...
       >>> mi.register_orphan_handler(orphan_handler, arg1, arg2, ...)
    """

    def _orphan_handler():
        handler(*args)

    if self._orphan_monitor is None:
        self._orphan_monitor = OrphanProcessMonitor(_orphan_handler)
        self._orphan_monitor.start()

register_teardown_handler(handler, *args)

Register teardown signal handler.

Parameters:

Name Type Description Default
handler Callable

Teardown signal handler.

 required
args

Arguments to the handler.

 ()

Examples:

>>> mi = ModularInput(...)
>>> def teardown_handler(arg1, arg2, ...):
>>>     ...
>>> mi.register_teardown_handler(teardown_handler, arg1, arg2, ...)
Source code in solnlib/modular_input/modular_input.py 
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
def register_teardown_handler(self, handler: Callable, *args):
    """Register teardown signal handler.

    Arguments:
        handler: Teardown signal handler.
        args: Arguments to the handler.

    Examples:
       >>> mi = ModularInput(...)
       >>> def teardown_handler(arg1, arg2, ...):
       >>>     ...
       >>> mi.register_teardown_handler(teardown_handler, arg1, arg2, ...)
    """

    def _teardown_handler(signum, frame):
        handler(*args)

    utils.handle_teardown_signals(_teardown_handler)

ModularInputException

Bases: Exception

Exception for ModularInput class.

Source code in solnlib/modular_input/modular_input.py 
41
42
43
44
class ModularInputException(Exception):
    """Exception for ModularInput class."""

    pass

Ended: modular_input

acl.py

This module contains interfaces that support CRUD operations on ACL.

__all__ = ['ACLException', 'ACLManager'] module-attribute

ACLException

Bases: Exception

Exception raised by ACLManager.

Source code in solnlib/acl.py 
30
31
32
33
class ACLException(Exception):
    """Exception raised by ACLManager."""

    pass

ACLManager

ACL manager.

Examples:

>>> import solnlib.acl as sacl
>>> saclm = sacl.ACLManager(session_key, 'Splunk_TA_test')
>>> saclm.get('data/transforms/extractions')
>>> saclm.update('data/transforms/extractions/_acl',
                 perms_read=['*'], perms_write=['*'])
Source code in solnlib/acl.py 
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
class ACLManager:
    """ACL manager.

    Examples:
       >>> import solnlib.acl as sacl
       >>> saclm = sacl.ACLManager(session_key, 'Splunk_TA_test')
       >>> saclm.get('data/transforms/extractions')
       >>> saclm.update('data/transforms/extractions/_acl',
                        perms_read=['*'], perms_write=['*'])
    """

    def __init__(
        self,
        session_key: str,
        app: str,
        owner: str = "nobody",
        scheme: str = None,
        host: str = None,
        port: int = None,
        **context: dict
    ):
        """Initializes ACLManager.

        Arguments:
            session_key: Splunk access token.
            app: App name of namespace.
            owner: (optional) Owner of namespace, default is `nobody`.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            context: Other configurations for Splunk rest client.
        """
        self._rest_client = rest_client.SplunkRestClient(
            session_key,
            app,
            owner=owner,
            scheme=scheme,
            host=host,
            port=port,
            **context
        )

    @retry(exceptions=[binding.HTTPError])
    def get(self, path: str) -> dict:
        """Get ACL of  /servicesNS/{`owner`}/{`app`}/{`path`}.

        Arguments:
            path: Path of ACL relative to /servicesNS/{`owner`}/{`app`}

        Returns:
            A dict contains ACL.

        Raises:
            ACLException: If `path` is invalid.

        Examples:
           >>> aclm = acl.ACLManager(session_key, 'Splunk_TA_test')
           >>> perms = aclm.get('data/transforms/extractions/_acl')
        """

        try:
            content = self._rest_client.get(path, output_mode="json").body.read()
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            raise ACLException("Invalid endpoint: %s.", path)

        return json.loads(content)["entry"][0]["acl"]

    @retry(exceptions=[binding.HTTPError])
    def update(
        self,
        path: str,
        owner: str = None,
        perms_read: List = None,
        perms_write: List = None,
    ) -> dict:
        """Update ACL of /servicesNS/{`owner`}/{`app`}/{`path`}.

        If the ACL is per-entity (ends in /acl), owner can be reassigned. If
        the acl is endpoint-level (ends in _acl), owner will be ignored. The
        'sharing' setting is always retrieved from the current.

        Arguments:
            path: Path of ACL relative to /servicesNS/{owner}/{app}. MUST
                end with /acl or /_acl indicating whether the permission is applied
                at the per-entity level or endpoint level respectively.
            owner: (optional) New owner of ACL, default is `nobody`.
            perms_read: (optional) List of roles (['*'] for all roles). If
                unspecified we will POST with current (if available) perms.read,
                default is None.
            perms_write: (optional) List of roles (['*'] for all roles). If
                unspecified we will POST with current (if available) perms.write,
                default is None.

        Returns:
            A dict contains ACL after update.

        Raises:
            ACLException: If `path` is invalid.

        Examples:
           >>> aclm = acl.ACLManager(session_key, 'Splunk_TA_test')
           >>> perms = aclm.update('data/transforms/extractions/_acl',
                                   perms_read=['admin'], perms_write=['admin'])
        """

        if not path.endswith("/acl") and not path.endswith("/_acl"):
            raise ACLException(
                "Invalid endpoint: %s, must end with /acl or /_acl." % path
            )

        curr_acl = self.get(path)

        postargs = {}
        if perms_read:
            postargs["perms.read"] = ",".join(perms_read)
        else:
            curr_read = curr_acl["perms"].get("read", [])
            if curr_read:
                postargs["perms.read"] = ",".join(curr_read)

        if perms_write:
            postargs["perms.write"] = ",".join(perms_write)
        else:
            curr_write = curr_acl["perms"].get("write", [])
            if curr_write:
                postargs["perms.write"] = ",".join(curr_write)

        if path.endswith("/acl"):
            # Allow ownership to be reset only at entity level.
            postargs["owner"] = owner or curr_acl["owner"]

        postargs["sharing"] = curr_acl["sharing"]

        try:
            content = self._rest_client.post(
                path, body=binding._encode(**postargs), output_mode="json"
            ).body.read()
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            raise ACLException("Invalid endpoint: %s.", path)

        return json.loads(content)["entry"][0]["acl"]

__init__(session_key, app, owner='nobody', scheme=None, host=None, port=None, **context)

Initializes ACLManager.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner str

(optional) Owner of namespace, default is nobody.

 'nobody'
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}
Source code in solnlib/acl.py 
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
def __init__(
    self,
    session_key: str,
    app: str,
    owner: str = "nobody",
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict
):
    """Initializes ACLManager.

    Arguments:
        session_key: Splunk access token.
        app: App name of namespace.
        owner: (optional) Owner of namespace, default is `nobody`.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.
    """
    self._rest_client = rest_client.SplunkRestClient(
        session_key,
        app,
        owner=owner,
        scheme=scheme,
        host=host,
        port=port,
        **context
    )

get(path)

Get ACL of /servicesNS/{owner}/{app}/{path}.

Parameters:

Name Type Description Default
path str

Path of ACL relative to /servicesNS/{owner}/{app}

 required

Returns:

Type Description
dict

A dict contains ACL.

Raises:

Type Description
ACLException

If path is invalid.

Examples:

>>> aclm = acl.ACLManager(session_key, 'Splunk_TA_test')
>>> perms = aclm.get('data/transforms/extractions/_acl')
Source code in solnlib/acl.py 
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
@retry(exceptions=[binding.HTTPError])
def get(self, path: str) -> dict:
    """Get ACL of  /servicesNS/{`owner`}/{`app`}/{`path`}.

    Arguments:
        path: Path of ACL relative to /servicesNS/{`owner`}/{`app`}

    Returns:
        A dict contains ACL.

    Raises:
        ACLException: If `path` is invalid.

    Examples:
       >>> aclm = acl.ACLManager(session_key, 'Splunk_TA_test')
       >>> perms = aclm.get('data/transforms/extractions/_acl')
    """

    try:
        content = self._rest_client.get(path, output_mode="json").body.read()
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise ACLException("Invalid endpoint: %s.", path)

    return json.loads(content)["entry"][0]["acl"]

update(path, owner=None, perms_read=None, perms_write=None)

Update ACL of /servicesNS/{owner}/{app}/{path}.

If the ACL is per-entity (ends in /acl), owner can be reassigned. If the acl is endpoint-level (ends in _acl), owner will be ignored. The ‘sharing’ setting is always retrieved from the current.

Parameters:

Name Type Description Default
path str

Path of ACL relative to /servicesNS/{owner}/{app}. MUST end with /acl or /_acl indicating whether the permission is applied at the per-entity level or endpoint level respectively.

 required
owner str

(optional) New owner of ACL, default is nobody.

 None
perms_read List

(optional) List of roles ([‘*’] for all roles). If unspecified we will POST with current (if available) perms.read, default is None.

 None
perms_write List

(optional) List of roles ([‘*’] for all roles). If unspecified we will POST with current (if available) perms.write, default is None.

 None

Returns:

Type Description
dict

A dict contains ACL after update.

Raises:

Type Description
ACLException

If path is invalid.

Examples:

>>> aclm = acl.ACLManager(session_key, 'Splunk_TA_test')
>>> perms = aclm.update('data/transforms/extractions/_acl',
                        perms_read=['admin'], perms_write=['admin'])
Source code in solnlib/acl.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
@retry(exceptions=[binding.HTTPError])
def update(
    self,
    path: str,
    owner: str = None,
    perms_read: List = None,
    perms_write: List = None,
) -> dict:
    """Update ACL of /servicesNS/{`owner`}/{`app`}/{`path`}.

    If the ACL is per-entity (ends in /acl), owner can be reassigned. If
    the acl is endpoint-level (ends in _acl), owner will be ignored. The
    'sharing' setting is always retrieved from the current.

    Arguments:
        path: Path of ACL relative to /servicesNS/{owner}/{app}. MUST
            end with /acl or /_acl indicating whether the permission is applied
            at the per-entity level or endpoint level respectively.
        owner: (optional) New owner of ACL, default is `nobody`.
        perms_read: (optional) List of roles (['*'] for all roles). If
            unspecified we will POST with current (if available) perms.read,
            default is None.
        perms_write: (optional) List of roles (['*'] for all roles). If
            unspecified we will POST with current (if available) perms.write,
            default is None.

    Returns:
        A dict contains ACL after update.

    Raises:
        ACLException: If `path` is invalid.

    Examples:
       >>> aclm = acl.ACLManager(session_key, 'Splunk_TA_test')
       >>> perms = aclm.update('data/transforms/extractions/_acl',
                               perms_read=['admin'], perms_write=['admin'])
    """

    if not path.endswith("/acl") and not path.endswith("/_acl"):
        raise ACLException(
            "Invalid endpoint: %s, must end with /acl or /_acl." % path
        )

    curr_acl = self.get(path)

    postargs = {}
    if perms_read:
        postargs["perms.read"] = ",".join(perms_read)
    else:
        curr_read = curr_acl["perms"].get("read", [])
        if curr_read:
            postargs["perms.read"] = ",".join(curr_read)

    if perms_write:
        postargs["perms.write"] = ",".join(perms_write)
    else:
        curr_write = curr_acl["perms"].get("write", [])
        if curr_write:
            postargs["perms.write"] = ",".join(curr_write)

    if path.endswith("/acl"):
        # Allow ownership to be reset only at entity level.
        postargs["owner"] = owner or curr_acl["owner"]

    postargs["sharing"] = curr_acl["sharing"]

    try:
        content = self._rest_client.post(
            path, body=binding._encode(**postargs), output_mode="json"
        ).body.read()
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise ACLException("Invalid endpoint: %s.", path)

    return json.loads(content)["entry"][0]["acl"]

credentials.py

This module contains Splunk credential related interfaces.

__all__ = ['CredentialException', 'CredentialNotExistException', 'CredentialManager', 'get_session_key'] module-attribute

CredentialException

Bases: Exception

General exception regarding credentials.

Source code in solnlib/credentials.py 
39
40
41
42
class CredentialException(Exception):
    """General exception regarding credentials."""

    pass

CredentialManager

Credential manager.

Examples:

>>> from solnlib import credentials
>>> cm = credentials.CredentialManager(session_key,
                                       'Splunk_TA_test',
                                       realm='realm_test')
Source code in solnlib/credentials.py 
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
class CredentialManager:
    """Credential manager.

    Examples:
       >>> from solnlib import credentials
       >>> cm = credentials.CredentialManager(session_key,
                                              'Splunk_TA_test',
                                              realm='realm_test')
    """

    # Splunk can only encrypt string with length <=255
    SPLUNK_CRED_LEN_LIMIT = 255

    # Splunk credential separator
    SEP = "``splunk_cred_sep``"

    # Splunk credential end mark
    END_MARK = (
        "``splunk_cred_sep``S``splunk_cred_sep``P``splunk_cred_sep``L``splunk_cred_sep``"
        "U``splunk_cred_sep``N``splunk_cred_sep``K``splunk_cred_sep``"
    )

    def __init__(
        self,
        session_key: str,
        app: str,
        owner: str = "nobody",
        realm: str = None,
        scheme: str = None,
        host: str = None,
        port: int = None,
        **context: dict,
    ):
        """Initializes CredentialManager.

        Arguments:
            session_key: Splunk access token.
            app: App name of namespace.
            owner: (optional) Owner of namespace, default is `nobody`.
            realm: (optional) Realm of credential, default is None.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            context: Other configurations for Splunk rest client.
        """
        self._realm = realm
        self.service = rest_client.SplunkRestClient(
            session_key,
            app,
            owner=owner,
            scheme=scheme,
            host=host,
            port=port,
            **context,
        )
        self._storage_passwords = self.service.storage_passwords

    @retry(exceptions=[binding.HTTPError])
    def get_password(self, user: str) -> str:
        """Get password.

        Arguments:
            user: User name.

        Returns:
            Clear user password.

        Raises:
            CredentialNotExistException: If password for 'realm:user' doesn't exist.

        Examples:
           >>> from solnlib import credentials
           >>> cm = credentials.CredentialManager(session_key,
                                                  'Splunk_TA_test',
                                                  realm='realm_test')
           >>> cm.get_password('testuser2')
        """
        if self._realm is not None:
            passwords = self.get_clear_passwords_in_realm()
        else:
            passwords = self.get_clear_passwords()
        for password in passwords:
            if password["username"] == user and password["realm"] == self._realm:
                return password["clear_password"]

        raise CredentialNotExistException(
            f"Failed to get password of realm={self._realm}, user={user}."
        )

    @retry(exceptions=[binding.HTTPError])
    def set_password(self, user: str, password: str):
        """Set password.

        Arguments:
            user: User name.
            password: User password.

        Examples:
           >>> from solnlib import credentials
           >>> cm = credentials.CredentialManager(session_key,
                                                  'Splunk_TA_test',
                                                  realm='realm_test')
           >>> cm.set_password('testuser1', 'password1')
        """
        length = 0
        index = 1
        while length < len(password):
            curr_str = password[
                length : length + self.SPLUNK_CRED_LEN_LIMIT  # noqa: E203
            ]
            partial_user = self.SEP.join([user, str(index)])
            self._update_password(partial_user, curr_str)
            length += self.SPLUNK_CRED_LEN_LIMIT
            index += 1

        # Append another stanza to mark the end of the password
        partial_user = self.SEP.join([user, str(index)])
        self._update_password(partial_user, self.END_MARK)

    @retry(exceptions=[binding.HTTPError])
    def _update_password(self, user: str, password: str):
        """Update password.

        Arguments:
            user: User name.
            password: User password.

        Examples:
           >>> from solnlib import credentials
           >>> cm = credentials.CredentialManager(session_key,
                                                  'Splunk_TA_test',
                                                  realm='realm_test')
           >>> cm._update_password('testuser1', 'password1')
        """
        try:
            self._storage_passwords.create(password, user, self._realm)
        except binding.HTTPError as ex:
            if ex.status == 409:
                if self._realm is not None:
                    passwords = self.get_raw_passwords_in_realm()
                else:
                    passwords = self.get_raw_passwords()
                for pwd_stanza in passwords:
                    if pwd_stanza.realm == self._realm and pwd_stanza.username == user:
                        pwd_stanza.update(password=password)
                        return
                raise ValueError(
                    f"Can not get the password object for realm: {self._realm} user: {user}"
                )
            else:
                raise ex

    @retry(exceptions=[binding.HTTPError])
    def delete_password(self, user: str):
        """Delete password.

        Arguments:
            user: User name.

        Raises:
             CredentialNotExistException: If password of realm:user doesn't exist.

        Examples:
           >>> from solnlib import credentials
           >>> cm = credentials.CredentialManager(session_key,
                                                  'Splunk_TA_test',
                                                  realm='realm_test')
           >>> cm.delete_password('testuser1')
        """
        if self._realm is not None:
            passwords = self.get_raw_passwords_in_realm()
        else:
            passwords = self.get_raw_passwords()
        deleted = False
        ent_pattern = re.compile(
            r"({}{}\d+)".format(user.replace("\\", "\\\\"), self.SEP)
        )
        for password in passwords:
            match = (user == password.username) or ent_pattern.match(password.username)
            if match and password.realm == self._realm:
                password.delete()
                deleted = True

        if not deleted:
            raise CredentialNotExistException(
                f"Failed to delete password of realm={self._realm}, user={user}"
            )

    def get_raw_passwords(self) -> List[client.StoragePassword]:
        """Returns all passwords in the "raw" format."""
        warnings.warn(
            "Please pass realm to the CredentialManager, "
            "so it can utilize get_raw_passwords_in_realm method instead."
        )
        return self._storage_passwords.list(count=-1)

    def get_raw_passwords_in_realm(self) -> List[client.StoragePassword]:
        """Returns all passwords within the realm in the "raw" format."""
        if self._realm is None:
            raise ValueError("No realm was specified")
        return self._storage_passwords.list(count=-1, search=f"realm={self._realm}")

    def get_clear_passwords(self) -> List[Dict[str, str]]:
        """Returns all passwords in the "clear" format."""
        warnings.warn(
            "Please pass realm to the CredentialManager, "
            "so it can utilize get_clear_passwords_in_realm method instead."
        )
        raw_passwords = self.get_raw_passwords()
        return self._get_clear_passwords(raw_passwords)

    def get_clear_passwords_in_realm(self) -> List[Dict[str, str]]:
        """Returns all passwords within the realm in the "clear" format."""
        if self._realm is None:
            raise ValueError("No realm was specified")
        raw_passwords = self.get_raw_passwords_in_realm()
        return self._get_clear_passwords(raw_passwords)

    def _get_all_passwords_in_realm(self) -> List[client.StoragePassword]:
        warnings.warn(
            "_get_all_passwords_in_realm is deprecated, "
            "please use get_raw_passwords_in_realm instead.",
            stacklevel=2,
        )
        if self._realm:
            all_passwords = self._storage_passwords.list(
                count=-1, search=f"realm={self._realm}"
            )
        else:
            all_passwords = self._storage_passwords.list(count=-1, search="")
        return all_passwords

    def _get_clear_passwords(
        self, passwords: List[client.StoragePassword]
    ) -> List[Dict[str, str]]:
        results = {}
        ptn = re.compile(rf"(.+){self.SEP}(\d+)")
        for password in passwords:
            match = ptn.match(password.name)
            if match:
                actual_name = match.group(1) + ":"
                index = int(match.group(2))
                if actual_name in results:
                    exist_stanza = results[actual_name]
                else:
                    exist_stanza = {}
                    exist_stanza["name"] = actual_name
                    exist_stanza["realm"] = password.realm
                    exist_stanza["username"] = password.username.split(self.SEP)[0]
                    exist_stanza["clears"] = {}
                    results[actual_name] = exist_stanza

                exist_stanza["clears"][index] = password.clear_password

        # Backward compatibility
        # To deal with the password with only one stanza which is generated by the old version.
        for password in passwords:
            match = ptn.match(password.name)
            if (not match) and (password.name not in results):
                results[password.name] = {
                    "name": password.name,
                    "realm": password.realm,
                    "username": password.username,
                    "clear_password": password.clear_password,
                }

        # Merge password by index
        for name, values in list(results.items()):
            field_clear = values.get("clears")
            if field_clear:
                clear_password = ""
                for index in sorted(field_clear.keys()):
                    if field_clear[index] != self.END_MARK:
                        clear_password += field_clear[index]
                    else:
                        break
                values["clear_password"] = clear_password

                del values["clears"]

        return list(results.values())

    @retry(exceptions=[binding.HTTPError])
    def _get_all_passwords(self) -> List[Dict[str, str]]:
        warnings.warn(
            "_get_all_passwords is deprecated, "
            "please use get_all_passwords_in_realm instead.",
            stacklevel=2,
        )
        passwords = self._storage_passwords.list(count=-1)
        return self._get_clear_passwords(passwords)

END_MARK = '``splunk_cred_sep``S``splunk_cred_sep``P``splunk_cred_sep``L``splunk_cred_sep``U``splunk_cred_sep``N``splunk_cred_sep``K``splunk_cred_sep``' instance-attribute class-attribute

SEP = '``splunk_cred_sep``' instance-attribute class-attribute

SPLUNK_CRED_LEN_LIMIT = 255 instance-attribute class-attribute

service = rest_client.SplunkRestClient(session_key, app, owner=owner, scheme=scheme, host=host, port=port, None=context) instance-attribute

__init__(session_key, app, owner='nobody', realm=None, scheme=None, host=None, port=None, **context)

Initializes CredentialManager.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner str

(optional) Owner of namespace, default is nobody.

 'nobody'
realm str

(optional) Realm of credential, default is None.

 None
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}
Source code in solnlib/credentials.py 
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
def __init__(
    self,
    session_key: str,
    app: str,
    owner: str = "nobody",
    realm: str = None,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
):
    """Initializes CredentialManager.

    Arguments:
        session_key: Splunk access token.
        app: App name of namespace.
        owner: (optional) Owner of namespace, default is `nobody`.
        realm: (optional) Realm of credential, default is None.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.
    """
    self._realm = realm
    self.service = rest_client.SplunkRestClient(
        session_key,
        app,
        owner=owner,
        scheme=scheme,
        host=host,
        port=port,
        **context,
    )
    self._storage_passwords = self.service.storage_passwords

delete_password(user)

Delete password.

Parameters:

Name Type Description Default
user str

User name.

 required

Raises:

Type Description
CredentialNotExistException

If password of realm:user doesn’t exist.

Examples:

>>> from solnlib import credentials
>>> cm = credentials.CredentialManager(session_key,
                                       'Splunk_TA_test',
                                       realm='realm_test')
>>> cm.delete_password('testuser1')
Source code in solnlib/credentials.py 
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
@retry(exceptions=[binding.HTTPError])
def delete_password(self, user: str):
    """Delete password.

    Arguments:
        user: User name.

    Raises:
         CredentialNotExistException: If password of realm:user doesn't exist.

    Examples:
       >>> from solnlib import credentials
       >>> cm = credentials.CredentialManager(session_key,
                                              'Splunk_TA_test',
                                              realm='realm_test')
       >>> cm.delete_password('testuser1')
    """
    if self._realm is not None:
        passwords = self.get_raw_passwords_in_realm()
    else:
        passwords = self.get_raw_passwords()
    deleted = False
    ent_pattern = re.compile(
        r"({}{}\d+)".format(user.replace("\\", "\\\\"), self.SEP)
    )
    for password in passwords:
        match = (user == password.username) or ent_pattern.match(password.username)
        if match and password.realm == self._realm:
            password.delete()
            deleted = True

    if not deleted:
        raise CredentialNotExistException(
            f"Failed to delete password of realm={self._realm}, user={user}"
        )

get_clear_passwords()

Returns all passwords in the “clear” format.

Source code in solnlib/credentials.py 
253
254
255
256
257
258
259
260
def get_clear_passwords(self) -> List[Dict[str, str]]:
    """Returns all passwords in the "clear" format."""
    warnings.warn(
        "Please pass realm to the CredentialManager, "
        "so it can utilize get_clear_passwords_in_realm method instead."
    )
    raw_passwords = self.get_raw_passwords()
    return self._get_clear_passwords(raw_passwords)

get_clear_passwords_in_realm()

Returns all passwords within the realm in the “clear” format.

Source code in solnlib/credentials.py 
262
263
264
265
266
267
def get_clear_passwords_in_realm(self) -> List[Dict[str, str]]:
    """Returns all passwords within the realm in the "clear" format."""
    if self._realm is None:
        raise ValueError("No realm was specified")
    raw_passwords = self.get_raw_passwords_in_realm()
    return self._get_clear_passwords(raw_passwords)

get_password(user)

Get password.

Parameters:

Name Type Description Default
user str

User name.

 required

Returns:

Type Description
str

Clear user password.

Raises:

Type Description
CredentialNotExistException

If password for ‘realm:user’ doesn’t exist.

Examples:

>>> from solnlib import credentials
>>> cm = credentials.CredentialManager(session_key,
                                       'Splunk_TA_test',
                                       realm='realm_test')
>>> cm.get_password('testuser2')
Source code in solnlib/credentials.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
@retry(exceptions=[binding.HTTPError])
def get_password(self, user: str) -> str:
    """Get password.

    Arguments:
        user: User name.

    Returns:
        Clear user password.

    Raises:
        CredentialNotExistException: If password for 'realm:user' doesn't exist.

    Examples:
       >>> from solnlib import credentials
       >>> cm = credentials.CredentialManager(session_key,
                                              'Splunk_TA_test',
                                              realm='realm_test')
       >>> cm.get_password('testuser2')
    """
    if self._realm is not None:
        passwords = self.get_clear_passwords_in_realm()
    else:
        passwords = self.get_clear_passwords()
    for password in passwords:
        if password["username"] == user and password["realm"] == self._realm:
            return password["clear_password"]

    raise CredentialNotExistException(
        f"Failed to get password of realm={self._realm}, user={user}."
    )

get_raw_passwords()

Returns all passwords in the “raw” format.

Source code in solnlib/credentials.py 
239
240
241
242
243
244
245
def get_raw_passwords(self) -> List[client.StoragePassword]:
    """Returns all passwords in the "raw" format."""
    warnings.warn(
        "Please pass realm to the CredentialManager, "
        "so it can utilize get_raw_passwords_in_realm method instead."
    )
    return self._storage_passwords.list(count=-1)

get_raw_passwords_in_realm()

Returns all passwords within the realm in the “raw” format.

Source code in solnlib/credentials.py 
247
248
249
250
251
def get_raw_passwords_in_realm(self) -> List[client.StoragePassword]:
    """Returns all passwords within the realm in the "raw" format."""
    if self._realm is None:
        raise ValueError("No realm was specified")
    return self._storage_passwords.list(count=-1, search=f"realm={self._realm}")

set_password(user, password)

Set password.

Parameters:

Name Type Description Default
user str

User name.

 required
password str

User password.

 required

Examples:

>>> from solnlib import credentials
>>> cm = credentials.CredentialManager(session_key,
                                       'Splunk_TA_test',
                                       realm='realm_test')
>>> cm.set_password('testuser1', 'password1')
Source code in solnlib/credentials.py 
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
@retry(exceptions=[binding.HTTPError])
def set_password(self, user: str, password: str):
    """Set password.

    Arguments:
        user: User name.
        password: User password.

    Examples:
       >>> from solnlib import credentials
       >>> cm = credentials.CredentialManager(session_key,
                                              'Splunk_TA_test',
                                              realm='realm_test')
       >>> cm.set_password('testuser1', 'password1')
    """
    length = 0
    index = 1
    while length < len(password):
        curr_str = password[
            length : length + self.SPLUNK_CRED_LEN_LIMIT  # noqa: E203
        ]
        partial_user = self.SEP.join([user, str(index)])
        self._update_password(partial_user, curr_str)
        length += self.SPLUNK_CRED_LEN_LIMIT
        index += 1

    # Append another stanza to mark the end of the password
    partial_user = self.SEP.join([user, str(index)])
    self._update_password(partial_user, self.END_MARK)

CredentialNotExistException

Bases: Exception

Exception is raised when credentials do not exist.

Source code in solnlib/credentials.py 
45
46
47
48
class CredentialNotExistException(Exception):
    """Exception is raised when credentials do not exist."""

    pass

get_session_key(username, password, scheme=None, host=None, port=None, **context)

Get splunkd access token.

Parameters:

Name Type Description Default
username str

The Splunk account username, which is used to authenticate the Splunk instance.

 required
password str

The Splunk account password.

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}

Returns:

Type Description
str

Splunk session key.

Raises:

Type Description
CredentialException

If username/password are invalid.
ValueError

if scheme, host or port are invalid.

Examples:

>>> get_session_key('user', 'password')
Source code in solnlib/credentials.py 
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
@retry(exceptions=[binding.HTTPError])
def get_session_key(
    username: str,
    password: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
) -> str:
    """Get splunkd access token.

    Arguments:
        username: The Splunk account username, which is used to authenticate the Splunk instance.
        password: The Splunk account password.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Returns:
        Splunk session key.

    Raises:
        CredentialException: If username/password are invalid.
        ValueError: if scheme, host or port are invalid.

    Examples:
       >>> get_session_key('user', 'password')
    """
    validate_scheme_host_port(scheme, host, port)

    if any([scheme is None, host is None, port is None]):
        scheme, host, port = get_splunkd_access_info()

    uri = "{scheme}://{host}:{port}/{endpoint}".format(
        scheme=scheme, host=host, port=port, endpoint="services/auth/login"
    )
    _rest_client = rest_client.SplunkRestClient(
        None, "-", "nobody", scheme, host, port, **context
    )
    try:
        response = _rest_client.http.post(
            uri, username=username, password=password, output_mode="json"
        )
    except binding.HTTPError as e:
        if e.status != 401:
            raise

        raise CredentialException("Invalid username/password.")

    return json.loads(response.body.read())["sessionKey"]

conf_manager.py

This module contains simple interfaces for Splunk config file management, you can update/get/delete stanzas and encrypt/decrypt some fields of stanza automatically.

__all__ = ['ConfFile', 'ConfManager'] module-attribute

ConfFile

Configuration file.

Source code in solnlib/conf_manager.py 
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
class ConfFile:
    """Configuration file."""

    ENCRYPTED_TOKEN = "******"

    reserved_keys = ("userName", "appName")

    def __init__(
        self,
        name: str,
        conf: client.ConfigurationFile,
        session_key: str,
        app: str,
        owner: str = "nobody",
        scheme: str = None,
        host: str = None,
        port: int = None,
        realm: str = None,
        **context: dict,
    ):
        """Initializes ConfFile.

        Arguments:
            name: Configuration file name.
            conf: Configuration file object.
            session_key: Splunk access token.
            app: App name of namespace.
            owner: (optional) Owner of namespace, default is `nobody`.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            realm: (optional) Realm of credential, default is None.
            context: Other configurations for Splunk rest client.
        """
        self._name = name
        self._conf = conf
        self._session_key = session_key
        self._app = app
        self._owner = owner
        self._scheme = scheme
        self._host = host
        self._port = port
        self._context = context
        self._cred_manager = None
        # 'realm' is set to provided 'realm' argument otherwise as default
        # behaviour it is set to 'APP_NAME'.
        if realm is None:
            self._realm = self._app
        else:
            self._realm = realm

    @property
    @retry(exceptions=[binding.HTTPError])
    def _cred_mgr(self):
        if self._cred_manager is None:
            self._cred_manager = CredentialManager(
                self._session_key,
                self._app,
                owner=self._owner,
                realm=self._realm,
                scheme=self._scheme,
                host=self._host,
                port=self._port,
                **self._context,
            )

        return self._cred_manager

    def _filter_stanza(self, stanza):
        for k in self.reserved_keys:
            if k in stanza:
                del stanza[k]

        return stanza

    def _encrypt_stanza(self, stanza_name, stanza, encrypt_keys):
        if not encrypt_keys:
            return stanza

        encrypt_stanza_keys = [k for k in encrypt_keys if k in stanza]
        encrypt_fields = {key: stanza[key] for key in encrypt_stanza_keys}
        if not encrypt_fields:
            return stanza
        self._cred_mgr.set_password(stanza_name, json.dumps(encrypt_fields))

        for key in encrypt_stanza_keys:
            stanza[key] = self.ENCRYPTED_TOKEN

        return stanza

    def _decrypt_stanza(self, stanza_name, encrypted_stanza):
        encrypted_keys = [
            key
            for key in encrypted_stanza
            if encrypted_stanza[key] == self.ENCRYPTED_TOKEN
        ]
        if encrypted_keys:
            encrypted_fields = json.loads(self._cred_mgr.get_password(stanza_name))
            for key in encrypted_keys:
                encrypted_stanza[key] = encrypted_fields[key]

        return encrypted_stanza

    def _delete_stanza_creds(self, stanza_name):
        self._cred_mgr.delete_password(stanza_name)

    @retry(exceptions=[binding.HTTPError])
    def stanza_exist(self, stanza_name: str) -> bool:
        """Check whether stanza exists.

        Arguments:
            stanza_name: Stanza name.

        Returns:
            True if stanza exists else False.

        Examples:
           >>> from solnlib import conf_manager
           >>> cfm = conf_manager.ConfManager(session_key,
                                              'Splunk_TA_test')
           >>> conf = cfm.get_conf('test')
           >>> conf.stanza_exist('test_stanza')
        """

        try:
            self._conf.list(name=stanza_name)[0]
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            return False

        return True

    @retry(exceptions=[binding.HTTPError])
    def get(self, stanza_name: str, only_current_app: bool = False) -> dict:
        """Get stanza from configuration file.

        Result is like:

            {
                'disabled': '0',
                'eai:appName': 'solnlib_demo',
                'eai:userName': 'nobody',
                'k1': '1',
                'k2': '2'
            }

        Arguments:
            stanza_name: Stanza name.
            only_current_app: Only include current app.

        Returns:
            Stanza.

        Raises:
            ConfStanzaNotExistException: If stanza does not exist.

        Examples:
           >>> from solnlib import conf_manager
           >>> cfm = conf_manager.ConfManager(session_key,
                                              'Splunk_TA_test')
           >>> conf = cfm.get_conf('test')
           >>> conf.get('test_stanza')
        """

        try:
            if only_current_app:
                stanza_mgrs = self._conf.list(
                    search="eai:acl.app={} name={}".format(
                        self._app, stanza_name.replace("=", r"\=")
                    )
                )
            else:
                stanza_mgrs = self._conf.list(name=stanza_name)
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            raise ConfStanzaNotExistException(
                f"Stanza: {stanza_name} does not exist in {self._name}.conf"
            )

        if len(stanza_mgrs) == 0:
            raise ConfStanzaNotExistException(
                f"Stanza: {stanza_name} does not exist in {self._name}.conf"
            )

        stanza = self._decrypt_stanza(stanza_mgrs[0].name, stanza_mgrs[0].content)
        stanza["eai:access"] = stanza_mgrs[0].access
        stanza["eai:appName"] = stanza_mgrs[0].access.app
        return stanza

    @retry(exceptions=[binding.HTTPError])
    def get_all(self, only_current_app: bool = False) -> dict:
        """Get all stanzas from configuration file.

        Result is like:

            {
                'test':
                    {
                        'disabled': '0',
                        'eai:appName': 'solnlib_demo',
                        'eai:userName': 'nobody',
                        'k1': '1',
                        'k2': '2'
                    }
            }

        Arguments:
            only_current_app: Only include current app.

        Returns:
            Dict of stanzas.

        Examples:
           >>> from solnlib import conf_manager
           >>> cfm = conf_manager.ConfManager(session_key,
                                              'Splunk_TA_test')
           >>> conf = cfm.get_conf('test')
           >>> conf.get_all()
        """

        if only_current_app:
            stanza_mgrs = self._conf.list(search=f"eai:acl.app={self._app}")
        else:
            stanza_mgrs = self._conf.list()
        res = {}
        for stanza_mgr in stanza_mgrs:
            name = stanza_mgr.name
            key_values = self._decrypt_stanza(name, stanza_mgr.content)
            key_values["eai:access"] = stanza_mgr.access
            key_values["eai:appName"] = stanza_mgr.access.app
            res[name] = key_values
        return res

    @retry(exceptions=[binding.HTTPError])
    def update(self, stanza_name: str, stanza: dict, encrypt_keys: List[str] = None):
        """Update stanza.

        It will try to encrypt the credential automatically fist if
        encrypt_keys are not None else keep stanza untouched.

        Arguments:
            stanza_name: Stanza name.
            stanza: Stanza to update.
            encrypt_keys: Field names to encrypt.

        Examples:
           >>> from solnlib import conf_manager
           >>> cfm = conf_manager.ConfManager(session_key,
                                              'Splunk_TA_test')
           >>> conf = cfm.get_conf('test')
           >>> conf.update('test_stanza', {'k1': 1, 'k2': 2}, ['k1'])
        """

        stanza = self._filter_stanza(stanza)
        encrypted_stanza = self._encrypt_stanza(stanza_name, stanza, encrypt_keys)

        try:
            stanza_mgr = self._conf.list(name=stanza_name)[0]
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            stanza_mgr = self._conf.create(stanza_name)

        stanza_mgr.submit(encrypted_stanza)

    @retry(exceptions=[binding.HTTPError])
    def delete(self, stanza_name: str):
        """Delete stanza.

        Arguments:
            stanza_name: Stanza name to delete.

        Raises:
            ConfStanzaNotExistException: If stanza does not exist.

        Examples:
           >>> from solnlib import conf_manager
           >>> cfm = conf_manager.ConfManager(session_key,
                                              'Splunk_TA_test')
           >>> conf = cfm.get_conf('test')
           >>> conf.delete('test_stanza')
        """

        try:
            self._cred_mgr.delete_password(stanza_name)
        except CredentialNotExistException:
            pass

        try:
            self._conf.delete(stanza_name)
        except KeyError:
            logging.error(
                "Delete stanza: %s error: %s.", stanza_name, traceback.format_exc()
            )
            raise ConfStanzaNotExistException(
                f"Stanza: {stanza_name} does not exist in {self._name}.conf"
            )

    @retry(exceptions=[binding.HTTPError])
    def reload(self):
        """Reload configuration file.

        Examples:
           >>> from solnlib import conf_manager
           >>> cfm = conf_manager.ConfManager(session_key,
                                              'Splunk_TA_test')
           >>> conf = cfm.get_conf('test')
           >>> conf.reload()
        """

        self._conf.get("_reload")

ENCRYPTED_TOKEN = '******' instance-attribute class-attribute

reserved_keys = ('userName', 'appName') instance-attribute class-attribute

__init__(name, conf, session_key, app, owner='nobody', scheme=None, host=None, port=None, realm=None, **context)

Initializes ConfFile.

Parameters:

Name Type Description Default
name str

Configuration file name.

 required
conf client.ConfigurationFile

Configuration file object.

 required
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner str

(optional) Owner of namespace, default is nobody.

 'nobody'
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
realm str

(optional) Realm of credential, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}
Source code in solnlib/conf_manager.py 
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
def __init__(
    self,
    name: str,
    conf: client.ConfigurationFile,
    session_key: str,
    app: str,
    owner: str = "nobody",
    scheme: str = None,
    host: str = None,
    port: int = None,
    realm: str = None,
    **context: dict,
):
    """Initializes ConfFile.

    Arguments:
        name: Configuration file name.
        conf: Configuration file object.
        session_key: Splunk access token.
        app: App name of namespace.
        owner: (optional) Owner of namespace, default is `nobody`.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        realm: (optional) Realm of credential, default is None.
        context: Other configurations for Splunk rest client.
    """
    self._name = name
    self._conf = conf
    self._session_key = session_key
    self._app = app
    self._owner = owner
    self._scheme = scheme
    self._host = host
    self._port = port
    self._context = context
    self._cred_manager = None
    # 'realm' is set to provided 'realm' argument otherwise as default
    # behaviour it is set to 'APP_NAME'.
    if realm is None:
        self._realm = self._app
    else:
        self._realm = realm

delete(stanza_name)

Delete stanza.

Parameters:

Name Type Description Default
stanza_name str

Stanza name to delete.

 required

Raises:

Type Description
ConfStanzaNotExistException

If stanza does not exist.

Examples:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(session_key,
                                   'Splunk_TA_test')
>>> conf = cfm.get_conf('test')
>>> conf.delete('test_stanza')
Source code in solnlib/conf_manager.py 
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
@retry(exceptions=[binding.HTTPError])
def delete(self, stanza_name: str):
    """Delete stanza.

    Arguments:
        stanza_name: Stanza name to delete.

    Raises:
        ConfStanzaNotExistException: If stanza does not exist.

    Examples:
       >>> from solnlib import conf_manager
       >>> cfm = conf_manager.ConfManager(session_key,
                                          'Splunk_TA_test')
       >>> conf = cfm.get_conf('test')
       >>> conf.delete('test_stanza')
    """

    try:
        self._cred_mgr.delete_password(stanza_name)
    except CredentialNotExistException:
        pass

    try:
        self._conf.delete(stanza_name)
    except KeyError:
        logging.error(
            "Delete stanza: %s error: %s.", stanza_name, traceback.format_exc()
        )
        raise ConfStanzaNotExistException(
            f"Stanza: {stanza_name} does not exist in {self._name}.conf"
        )

get(stanza_name, only_current_app=False)

Get stanza from configuration file.

Result is like

{ ‘disabled’: ‘0’, ‘eai:appName’: ‘solnlib_demo’, ‘eai:userName’: ‘nobody’, ‘k1’: ‘1’, ‘k2’: ‘2’ }

Parameters:

Name Type Description Default
stanza_name str

Stanza name.

 required
only_current_app bool

Only include current app.

 False

Returns:

Type Description
dict

Stanza.

Raises:

Type Description
ConfStanzaNotExistException

If stanza does not exist.

Examples:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(session_key,
                                   'Splunk_TA_test')
>>> conf = cfm.get_conf('test')
>>> conf.get('test_stanza')
Source code in solnlib/conf_manager.py 
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
@retry(exceptions=[binding.HTTPError])
def get(self, stanza_name: str, only_current_app: bool = False) -> dict:
    """Get stanza from configuration file.

    Result is like:

        {
            'disabled': '0',
            'eai:appName': 'solnlib_demo',
            'eai:userName': 'nobody',
            'k1': '1',
            'k2': '2'
        }

    Arguments:
        stanza_name: Stanza name.
        only_current_app: Only include current app.

    Returns:
        Stanza.

    Raises:
        ConfStanzaNotExistException: If stanza does not exist.

    Examples:
       >>> from solnlib import conf_manager
       >>> cfm = conf_manager.ConfManager(session_key,
                                          'Splunk_TA_test')
       >>> conf = cfm.get_conf('test')
       >>> conf.get('test_stanza')
    """

    try:
        if only_current_app:
            stanza_mgrs = self._conf.list(
                search="eai:acl.app={} name={}".format(
                    self._app, stanza_name.replace("=", r"\=")
                )
            )
        else:
            stanza_mgrs = self._conf.list(name=stanza_name)
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise ConfStanzaNotExistException(
            f"Stanza: {stanza_name} does not exist in {self._name}.conf"
        )

    if len(stanza_mgrs) == 0:
        raise ConfStanzaNotExistException(
            f"Stanza: {stanza_name} does not exist in {self._name}.conf"
        )

    stanza = self._decrypt_stanza(stanza_mgrs[0].name, stanza_mgrs[0].content)
    stanza["eai:access"] = stanza_mgrs[0].access
    stanza["eai:appName"] = stanza_mgrs[0].access.app
    return stanza

get_all(only_current_app=False)

Get all stanzas from configuration file.

Result is like

{ ‘test’: { ‘disabled’: ‘0’, ‘eai:appName’: ‘solnlib_demo’, ‘eai:userName’: ‘nobody’, ‘k1’: ‘1’, ‘k2’: ‘2’ } }

Parameters:

Name Type Description Default
only_current_app bool

Only include current app.

 False

Returns:

Type Description
dict

Dict of stanzas.

Examples:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(session_key,
                                   'Splunk_TA_test')
>>> conf = cfm.get_conf('test')
>>> conf.get_all()
Source code in solnlib/conf_manager.py 
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
@retry(exceptions=[binding.HTTPError])
def get_all(self, only_current_app: bool = False) -> dict:
    """Get all stanzas from configuration file.

    Result is like:

        {
            'test':
                {
                    'disabled': '0',
                    'eai:appName': 'solnlib_demo',
                    'eai:userName': 'nobody',
                    'k1': '1',
                    'k2': '2'
                }
        }

    Arguments:
        only_current_app: Only include current app.

    Returns:
        Dict of stanzas.

    Examples:
       >>> from solnlib import conf_manager
       >>> cfm = conf_manager.ConfManager(session_key,
                                          'Splunk_TA_test')
       >>> conf = cfm.get_conf('test')
       >>> conf.get_all()
    """

    if only_current_app:
        stanza_mgrs = self._conf.list(search=f"eai:acl.app={self._app}")
    else:
        stanza_mgrs = self._conf.list()
    res = {}
    for stanza_mgr in stanza_mgrs:
        name = stanza_mgr.name
        key_values = self._decrypt_stanza(name, stanza_mgr.content)
        key_values["eai:access"] = stanza_mgr.access
        key_values["eai:appName"] = stanza_mgr.access.app
        res[name] = key_values
    return res

reload()

Reload configuration file.

Examples:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(session_key,
                                   'Splunk_TA_test')
>>> conf = cfm.get_conf('test')
>>> conf.reload()
Source code in solnlib/conf_manager.py 
348
349
350
351
352
353
354
355
356
357
358
359
360
@retry(exceptions=[binding.HTTPError])
def reload(self):
    """Reload configuration file.

    Examples:
       >>> from solnlib import conf_manager
       >>> cfm = conf_manager.ConfManager(session_key,
                                          'Splunk_TA_test')
       >>> conf = cfm.get_conf('test')
       >>> conf.reload()
    """

    self._conf.get("_reload")

stanza_exist(stanza_name)

Check whether stanza exists.

Parameters:

Name Type Description Default
stanza_name str

Stanza name.

 required

Returns:

Type Description
bool

True if stanza exists else False.

Examples:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(session_key,
                                   'Splunk_TA_test')
>>> conf = cfm.get_conf('test')
>>> conf.stanza_exist('test_stanza')
Source code in solnlib/conf_manager.py 
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
@retry(exceptions=[binding.HTTPError])
def stanza_exist(self, stanza_name: str) -> bool:
    """Check whether stanza exists.

    Arguments:
        stanza_name: Stanza name.

    Returns:
        True if stanza exists else False.

    Examples:
       >>> from solnlib import conf_manager
       >>> cfm = conf_manager.ConfManager(session_key,
                                          'Splunk_TA_test')
       >>> conf = cfm.get_conf('test')
       >>> conf.stanza_exist('test_stanza')
    """

    try:
        self._conf.list(name=stanza_name)[0]
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        return False

    return True

update(stanza_name, stanza, encrypt_keys=None)

Update stanza.

It will try to encrypt the credential automatically fist if encrypt_keys are not None else keep stanza untouched.

Parameters:

Name Type Description Default
stanza_name str

Stanza name.

 required
stanza dict

Stanza to update.

 required
encrypt_keys List[str]

Field names to encrypt.

 None

Examples:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(session_key,
                                   'Splunk_TA_test')
>>> conf = cfm.get_conf('test')
>>> conf.update('test_stanza', {'k1': 1, 'k2': 2}, ['k1'])
Source code in solnlib/conf_manager.py 
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
@retry(exceptions=[binding.HTTPError])
def update(self, stanza_name: str, stanza: dict, encrypt_keys: List[str] = None):
    """Update stanza.

    It will try to encrypt the credential automatically fist if
    encrypt_keys are not None else keep stanza untouched.

    Arguments:
        stanza_name: Stanza name.
        stanza: Stanza to update.
        encrypt_keys: Field names to encrypt.

    Examples:
       >>> from solnlib import conf_manager
       >>> cfm = conf_manager.ConfManager(session_key,
                                          'Splunk_TA_test')
       >>> conf = cfm.get_conf('test')
       >>> conf.update('test_stanza', {'k1': 1, 'k2': 2}, ['k1'])
    """

    stanza = self._filter_stanza(stanza)
    encrypted_stanza = self._encrypt_stanza(stanza_name, stanza, encrypt_keys)

    try:
        stanza_mgr = self._conf.list(name=stanza_name)[0]
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        stanza_mgr = self._conf.create(stanza_name)

    stanza_mgr.submit(encrypted_stanza)

ConfManager

Configuration file manager.

Examples:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(session_key,
                                  'Splunk_TA_test')

Examples:

If stanza in passwords.conf is formatted as below:

credential:__REST_CREDENTIAL__#Splunk_TA_test#configs/conf-CONF_FILENAME:STANZA_NAME``splunk_cred_sep``1:

>>> from solnlib import conf_manager
>>> cfm = conf_manager.ConfManager(
        session_key,
        'Splunk_TA_test',
        realm='__REST_CREDENTIAL__#Splunk_TA_test#configs/conf-CONF_FILENAME'
    )
Source code in solnlib/conf_manager.py 
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
class ConfManager:
    """Configuration file manager.

    Examples:

        >>> from solnlib import conf_manager
        >>> cfm = conf_manager.ConfManager(session_key,
                                          'Splunk_TA_test')

    Examples:
        If stanza in passwords.conf is formatted as below:

        `credential:__REST_CREDENTIAL__#Splunk_TA_test#configs/conf-CONF_FILENAME:STANZA_NAME``splunk_cred_sep``1:`

        >>> from solnlib import conf_manager
        >>> cfm = conf_manager.ConfManager(
                session_key,
                'Splunk_TA_test',
                realm='__REST_CREDENTIAL__#Splunk_TA_test#configs/conf-CONF_FILENAME'
            )
    """

    def __init__(
        self,
        session_key: str,
        app: str,
        owner: str = "nobody",
        scheme: str = None,
        host: str = None,
        port: int = None,
        realm: str = None,
        **context: dict,
    ):
        """Initializes ConfManager.

        Arguments:
            session_key: Splunk access token.
            app: App name of namespace.
            owner: (optional) Owner of namespace, default is `nobody`.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            realm: (optional) Realm of credential, default is None.
            context: Other configurations for Splunk rest client.
        """
        self._session_key = session_key
        self._app = app
        self._owner = owner
        self._scheme = scheme
        self._host = host
        self._port = port
        self._context = context
        self._rest_client = rest_client.SplunkRestClient(
            self._session_key,
            self._app,
            owner=self._owner,
            scheme=self._scheme,
            host=self._host,
            port=self._port,
            **self._context,
        )
        self._confs = None
        self._realm = realm

    @retry(exceptions=[binding.HTTPError])
    def get_conf(self, name: str, refresh: bool = False) -> ConfFile:
        """Get conf file.

        Arguments:
            name: Conf file name.
            refresh: (optional) Flag to refresh conf file list, default is False.

        Returns:
            Conf file object.

        Raises:
            ConfManagerException: If `conf_file` does not exist.
        """

        if self._confs is None or refresh:
            # Fix bug that can't pass `-` as app name.
            curr_app = self._rest_client.namespace.app
            self._rest_client.namespace.app = "dummy"
            self._confs = self._rest_client.confs
            self._rest_client.namespace.app = curr_app

        try:
            conf = self._confs[name]
        except KeyError:
            raise ConfManagerException(f"Config file: {name} does not exist.")

        return ConfFile(
            name,
            conf,
            self._session_key,
            self._app,
            self._owner,
            self._scheme,
            self._host,
            self._port,
            self._realm,
            **self._context,
        )

    @retry(exceptions=[binding.HTTPError])
    def create_conf(self, name: str) -> ConfFile:
        """Create conf file.

        Arguments:
            name: Conf file name.

        Returns:
            Conf file object.
        """

        if self._confs is None:
            self._confs = self._rest_client.confs

        conf = self._confs.create(name)
        return ConfFile(
            name,
            conf,
            self._session_key,
            self._app,
            self._owner,
            self._scheme,
            self._host,
            self._port,
            self._realm,
            **self._context,
        )

__init__(session_key, app, owner='nobody', scheme=None, host=None, port=None, realm=None, **context)

Initializes ConfManager.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner str

(optional) Owner of namespace, default is nobody.

 'nobody'
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
realm str

(optional) Realm of credential, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}
Source code in solnlib/conf_manager.py 
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
def __init__(
    self,
    session_key: str,
    app: str,
    owner: str = "nobody",
    scheme: str = None,
    host: str = None,
    port: int = None,
    realm: str = None,
    **context: dict,
):
    """Initializes ConfManager.

    Arguments:
        session_key: Splunk access token.
        app: App name of namespace.
        owner: (optional) Owner of namespace, default is `nobody`.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        realm: (optional) Realm of credential, default is None.
        context: Other configurations for Splunk rest client.
    """
    self._session_key = session_key
    self._app = app
    self._owner = owner
    self._scheme = scheme
    self._host = host
    self._port = port
    self._context = context
    self._rest_client = rest_client.SplunkRestClient(
        self._session_key,
        self._app,
        owner=self._owner,
        scheme=self._scheme,
        host=self._host,
        port=self._port,
        **self._context,
    )
    self._confs = None
    self._realm = realm

create_conf(name)

Create conf file.

Parameters:

Name Type Description Default
name str

Conf file name.

 required

Returns:

Type Description
ConfFile

Conf file object.
Source code in solnlib/conf_manager.py 
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
@retry(exceptions=[binding.HTTPError])
def create_conf(self, name: str) -> ConfFile:
    """Create conf file.

    Arguments:
        name: Conf file name.

    Returns:
        Conf file object.
    """

    if self._confs is None:
        self._confs = self._rest_client.confs

    conf = self._confs.create(name)
    return ConfFile(
        name,
        conf,
        self._session_key,
        self._app,
        self._owner,
        self._scheme,
        self._host,
        self._port,
        self._realm,
        **self._context,
    )

get_conf(name, refresh=False)

Get conf file.

Parameters:

Name Type Description Default
name str

Conf file name.

 required
refresh bool

(optional) Flag to refresh conf file list, default is False.

 False

Returns:

Type Description
ConfFile

Conf file object.

Raises:

Type Description
ConfManagerException

If conf_file does not exist.
Source code in solnlib/conf_manager.py 
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
@retry(exceptions=[binding.HTTPError])
def get_conf(self, name: str, refresh: bool = False) -> ConfFile:
    """Get conf file.

    Arguments:
        name: Conf file name.
        refresh: (optional) Flag to refresh conf file list, default is False.

    Returns:
        Conf file object.

    Raises:
        ConfManagerException: If `conf_file` does not exist.
    """

    if self._confs is None or refresh:
        # Fix bug that can't pass `-` as app name.
        curr_app = self._rest_client.namespace.app
        self._rest_client.namespace.app = "dummy"
        self._confs = self._rest_client.confs
        self._rest_client.namespace.app = curr_app

    try:
        conf = self._confs[name]
    except KeyError:
        raise ConfManagerException(f"Config file: {name} does not exist.")

    return ConfFile(
        name,
        conf,
        self._session_key,
        self._app,
        self._owner,
        self._scheme,
        self._host,
        self._port,
        self._realm,
        **self._context,
    )

get_log_level(*, logger, session_key, app_name, conf_name, log_stanza='logging', log_level_field='loglevel', default_log_level='INFO')

This function returns the log level for the addon from configuration file.

Parameters:

Name Type Description Default
logger logging.Logger

Logger.

 required
session_key str

Splunk access token.

 required
app_name str

Add-on name.

 required
conf_name str

Configuration file name where logging stanza is.

 required
log_stanza str

Logging stanza to define log_level_field and its value.

 'logging'
log_level_field str

Logging level field name under logging stanza.

 'loglevel'
default_log_level str

Default log level to return in case of errors.

 'INFO'

Returns:

Type Description
str

Log level defined under logging.log_level_field field in conf_name
str

file. In case of any error, default_log_level will be returned.

Examples:

>>> from solnlib import conf_manager
>>> log_level = conf_manager.get_log_level(
>>>     logger,
>>>     "session_key",
>>>     "ADDON_NAME",
>>>     "splunk_ta_addon_settings",
>>> )
Source code in solnlib/conf_manager.py 
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
def get_log_level(
    *,
    logger: logging.Logger,
    session_key: str,
    app_name: str,
    conf_name: str,
    log_stanza: str = "logging",
    log_level_field: str = "loglevel",
    default_log_level: str = "INFO",
) -> str:
    """This function returns the log level for the addon from configuration
    file.

    Arguments:
        logger: Logger.
        session_key: Splunk access token.
        app_name: Add-on name.
        conf_name: Configuration file name where logging stanza is.
        log_stanza: Logging stanza to define `log_level_field` and its value.
        log_level_field: Logging level field name under logging stanza.
        default_log_level: Default log level to return in case of errors.

    Returns:
        Log level defined under `logging.log_level_field` field in `conf_name`
        file. In case of any error, `default_log_level` will be returned.

    Examples:
        >>> from solnlib import conf_manager
        >>> log_level = conf_manager.get_log_level(
        >>>     logger,
        >>>     "session_key",
        >>>     "ADDON_NAME",
        >>>     "splunk_ta_addon_settings",
        >>> )
    """
    try:
        cfm = ConfManager(
            session_key,
            app_name,
            realm=f"__REST_CREDENTIAL__#{app_name}#configs/conf-{conf_name}",
        )
        conf = cfm.get_conf(conf_name)
    except ConfManagerException:
        logger.error(
            f"Failed to fetch configuration file {conf_name}, "
            f"taking {default_log_level} as log level."
        )
        return default_log_level
    try:
        logging_details = conf.get(log_stanza)
        return logging_details.get(log_level_field, default_log_level)
    except ConfStanzaNotExistException:
        logger.error(
            f'"logging" stanza does not exist under {conf_name}, '
            f"taking {default_log_level} as log level."
        )
        return default_log_level

get_proxy_dict(logger, session_key, app_name, conf_name, proxy_stanza='proxy', **kwargs)

This function returns the proxy settings for the addon from configuration file.

Parameters:

Name Type Description Default
logger logging.Logger

Logger.

 required
session_key str

Splunk access token.

 required
app_name str

Add-on name.

 required
conf_name str

Configuration file name where logging stanza is.

 required
proxy_stanza str

Proxy stanza that would contain the Proxy details

 'proxy'

Returns:

Type Description
Union[Dict[str, str], NoReturn]

A dictionary is returned with stanza details present in the file.
Union[Dict[str, str], NoReturn]

The keys related to eai are removed before returning.

Examples:

>>> from solnlib import conf_manager
>>> proxy_details = conf_manager.get_proxy_dict(
>>>     logger,
>>>     "session_key",
>>>     "ADDON_NAME",
>>>     "splunk_ta_addon_settings",
>>> )
Source code in solnlib/conf_manager.py 
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
def get_proxy_dict(
    logger: logging.Logger,
    session_key: str,
    app_name: str,
    conf_name: str,
    proxy_stanza: str = "proxy",
    **kwargs,
) -> Union[Dict[str, str], NoReturn]:
    """This function returns the proxy settings for the addon from
    configuration file.

    Arguments:
        logger: Logger.
        session_key: Splunk access token.
        app_name: Add-on name.
        conf_name: Configuration file name where logging stanza is.
        proxy_stanza: Proxy stanza that would contain the Proxy details
    Returns:
        A dictionary is returned with stanza details present in the file.
        The keys related to `eai` are removed before returning.

    Examples:
        >>> from solnlib import conf_manager
        >>> proxy_details = conf_manager.get_proxy_dict(
        >>>     logger,
        >>>     "session_key",
        >>>     "ADDON_NAME",
        >>>     "splunk_ta_addon_settings",
        >>> )
    """
    proxy_dict = {}
    try:
        cfm = ConfManager(
            session_key,
            app_name,
            realm=f"__REST_CREDENTIAL__#{app_name}#configs/conf-{conf_name}",
        )
        conf = cfm.get_conf(conf_name)
    except Exception:
        raise ConfManagerException(f"Failed to fetch configuration file '{conf_name}'.")
    else:
        try:
            proxy_dict = conf.get(proxy_stanza)
        except Exception:
            raise ConfStanzaNotExistException(
                f"Failed to fetch '{proxy_stanza}' from the configuration file '{conf_name}'. "
            )
        else:
            # remove the other fields that are added by ConfFile class
            proxy_dict.pop("disabled", None)
            proxy_dict.pop("eai:access", None)
            proxy_dict.pop("eai:appName", None)
            proxy_dict.pop("eai:userName", None)

            if "proxy_port" in kwargs:
                if not is_valid_port(proxy_dict.get(kwargs["proxy_port"])):
                    logger.error("Invalid proxy port provided.")
                    raise InvalidPortError("The provided port is not valid.")
            if "proxy_host" in kwargs:
                if not is_valid_hostname(proxy_dict.get(kwargs["proxy_host"])):
                    logger.error("Invalid proxy host provided.")
                    raise InvalidHostnameError("The provided hostname is not valid.")
    return proxy_dict

file_monitor.py

This module contains file monitoring class that can be used to check files change periodically and call callback function to handle properly when detecting files change.

__all__ = ['FileChangesChecker', 'FileMonitor'] module-attribute

FileChangesChecker

Files change checker.

Source code in solnlib/file_monitor.py 
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
class FileChangesChecker:
    """Files change checker."""

    def __init__(self, callback: Callable[[List[str]], Any], files: List):
        """Initializes FileChangesChecker.

        Arguments:
            callback: Callback function for files change.
            files: Files to be monitored with full path.
        """
        self._callback = callback
        self._files = files

        self.file_mtimes = {file_name: None for file_name in self._files}
        for k in self.file_mtimes:
            try:
                self.file_mtimes[k] = op.getmtime(k)
            except OSError:
                logging.debug(f"Getmtime for {k}, failed: {traceback.format_exc()}")

    def check_changes(self) -> bool:
        """Check files change.

        If some files are changed and callback function is not None, call
        callback function to handle files change.

        Returns:
            True if files changed else False
        """
        logging.debug(f"Checking files={self._files}")
        file_mtimes = self.file_mtimes
        changed_files = []
        for f, last_mtime in list(file_mtimes.items()):
            try:
                current_mtime = op.getmtime(f)
                if current_mtime != last_mtime:
                    file_mtimes[f] = current_mtime
                    changed_files.append(f)
                    logging.info(f"Detect {f} has changed", f)
            except OSError:
                pass
        if changed_files:
            if self._callback:
                self._callback(changed_files)
            return True
        return False

file_mtimes = {file_name: None for file_name in self._files} instance-attribute

__init__(callback, files)

Initializes FileChangesChecker.

Parameters:

Name Type Description Default
callback Callable[[List[str]], Any]

Callback function for files change.

 required
files List

Files to be monitored with full path.

 required
Source code in solnlib/file_monitor.py 
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
def __init__(self, callback: Callable[[List[str]], Any], files: List):
    """Initializes FileChangesChecker.

    Arguments:
        callback: Callback function for files change.
        files: Files to be monitored with full path.
    """
    self._callback = callback
    self._files = files

    self.file_mtimes = {file_name: None for file_name in self._files}
    for k in self.file_mtimes:
        try:
            self.file_mtimes[k] = op.getmtime(k)
        except OSError:
            logging.debug(f"Getmtime for {k}, failed: {traceback.format_exc()}")

check_changes()

Check files change.

If some files are changed and callback function is not None, call callback function to handle files change.

Returns:

Type Description
bool

True if files changed else False
Source code in solnlib/file_monitor.py 
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
def check_changes(self) -> bool:
    """Check files change.

    If some files are changed and callback function is not None, call
    callback function to handle files change.

    Returns:
        True if files changed else False
    """
    logging.debug(f"Checking files={self._files}")
    file_mtimes = self.file_mtimes
    changed_files = []
    for f, last_mtime in list(file_mtimes.items()):
        try:
            current_mtime = op.getmtime(f)
            if current_mtime != last_mtime:
                file_mtimes[f] = current_mtime
                changed_files.append(f)
                logging.info(f"Detect {f} has changed", f)
        except OSError:
            pass
    if changed_files:
        if self._callback:
            self._callback(changed_files)
        return True
    return False

FileMonitor

Files change monitor.

Monitor files change in a separated thread and call callback when there is files change.

Examples:

>>> import solnlib.file_monitor as fm
>>> fm = fm.FileMonitor(fm_callback, files_list, 5)
>>> fm.start()
Source code in solnlib/file_monitor.py 
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
class FileMonitor:
    """Files change monitor.

    Monitor files change in a separated thread and call callback
    when there is files change.

    Examples:
      >>> import solnlib.file_monitor as fm
      >>> fm = fm.FileMonitor(fm_callback, files_list, 5)
      >>> fm.start()
    """

    def __init__(
        self, callback: Callable[[List[str]], Any], files: List, interval: int = 1
    ):
        """Initializes FileMonitor.

        Arguments:
            callback: Callback for handling files change.
            files: Files to monitor.
            interval: Interval to check files change.
        """
        self._checker = FileChangesChecker(callback, files)
        self._thr = threading.Thread(target=self._do_monitor)
        self._thr.daemon = True
        self._interval = interval
        self._started = False

    def start(self):
        """Start file monitor.

        Start a background thread to monitor files change.
        """

        if self._started:
            return
        self._started = True

        self._thr.start()

    def stop(self):
        """Stop file monitor.

        Stop the background thread to monitor files change.
        """

        self._started = False

    def _do_monitor(self):
        while self._started:
            self._checker.check_changes()

            for _ in range(self._interval):
                if not self._started:
                    break
                time.sleep(1)

__init__(callback, files, interval=1)

Initializes FileMonitor.

Parameters:

Name Type Description Default
callback Callable[[List[str]], Any]

Callback for handling files change.

 required
files List

Files to monitor.

 required
interval int

Interval to check files change.

 1
Source code in solnlib/file_monitor.py 
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def __init__(
    self, callback: Callable[[List[str]], Any], files: List, interval: int = 1
):
    """Initializes FileMonitor.

    Arguments:
        callback: Callback for handling files change.
        files: Files to monitor.
        interval: Interval to check files change.
    """
    self._checker = FileChangesChecker(callback, files)
    self._thr = threading.Thread(target=self._do_monitor)
    self._thr.daemon = True
    self._interval = interval
    self._started = False

start()

Start file monitor.

Start a background thread to monitor files change.

Source code in solnlib/file_monitor.py 
106
107
108
109
110
111
112
113
114
115
116
def start(self):
    """Start file monitor.

    Start a background thread to monitor files change.
    """

    if self._started:
        return
    self._started = True

    self._thr.start()

stop()

Stop file monitor.

Stop the background thread to monitor files change.

Source code in solnlib/file_monitor.py 
118
119
120
121
122
123
124
def stop(self):
    """Stop file monitor.

    Stop the background thread to monitor files change.
    """

    self._started = False

hec_config.py

__all__ = ['HECConfig'] module-attribute

HECConfig

HTTP Event Collector configuration.

Source code in solnlib/hec_config.py 
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
class HECConfig:
    """HTTP Event Collector configuration."""

    input_type = "http"

    def __init__(
        self,
        session_key: str,
        scheme: str = None,
        host: str = None,
        port: int = None,
        **context: dict
    ):
        """Initializes HECConfig.

        Arguments:
            session_key: Splunk access token.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            context: Other configurations for Splunk rest client.
        """
        self._rest_client = rest_client.SplunkRestClient(
            session_key,
            "splunk_httpinput",
            scheme=scheme,
            host=host,
            port=port,
            **context
        )

    @retry(exceptions=[binding.HTTPError])
    def get_settings(self) -> dict:
        """Get http data input global settings.

        Returns:
            HTTP global settings, for example:

                {
                    'enableSSL': 1,
                    'disabled': 0,
                    'useDeploymentServer': 0,
                    'port': 8088
                }
        """

        return self._do_get_input(self.input_type).content

    @retry(exceptions=[binding.HTTPError])
    def update_settings(self, settings: dict):
        """Update http data input global settings.

        Arguments:
            settings: HTTP global settings.
        """

        res = self._do_get_input(self.input_type)
        res.update(**settings)

    @retry(exceptions=[binding.HTTPError])
    def create_input(self, name: str, stanza: dict) -> dict:
        """Create http data input.

        Arguments:
            name: HTTP data input name.
            stanza: Data input stanza content.

        Returns:
            Created input.

        Examples:
           >>> from solnlib.hec_config import HECConfig
           >>> hec = HECConfig(session_key)
           >>> hec.create_input('my_hec_data_input',
                                {'index': 'main', 'sourcetype': 'hec'})
        """

        res = self._rest_client.inputs.create(name, self.input_type, **stanza)
        return res.content

    @retry(exceptions=[binding.HTTPError])
    def update_input(self, name: str, stanza: dict):
        """Update http data input.

        It will create if the data input doesn't exist.

        Arguments:
            name: HTTP data input name.
            stanza: Data input stanza.

        Examples:
           >>> from solnlib import HEConfig
           >>> hec = HECConfig(session_key)
           >>> hec.update_input('my_hec_data_input',
                                {'index': 'main', 'sourcetype': 'hec2'})
        """

        res = self._do_get_input(name)
        if res is None:
            return self.create_input(name, stanza)
        res.update(**stanza)

    @retry(exceptions=[binding.HTTPError])
    def delete_input(self, name: str):
        """Delete http data input.

        Arguments:
            name: HTTP data input name.
        """

        try:
            self._rest_client.inputs.delete(name, self.input_type)
        except KeyError:
            pass

    @retry(exceptions=[binding.HTTPError])
    def get_input(self, name: str) -> dict:
        """Get http data input.

        Arguments:
            name: HTTP event collector data input name.

        Returns:
            HTTP event collector data input config dict.
        """

        res = self._do_get_input(name)
        if res:
            return res.content
        else:
            return None

    def _do_get_input(self, name):
        try:
            return self._rest_client.inputs[(name, self.input_type)]
        except KeyError:
            return None

    @retry(exceptions=[binding.HTTPError])
    def get_limits(self) -> dict:
        """Get HTTP input limits.

        Returns:
            HTTP input limits.
        """

        return self._rest_client.confs["limits"]["http_input"].content

    @retry(exceptions=[binding.HTTPError])
    def set_limits(self, limits: dict):
        """Set HTTP input limits.

        Arguments:
            limits: HTTP input limits.
        """

        res = self._rest_client.confs["limits"]["http_input"]
        res.submit(limits)

input_type = 'http' instance-attribute class-attribute

__init__(session_key, scheme=None, host=None, port=None, **context)

Initializes HECConfig.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}
Source code in solnlib/hec_config.py 
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
def __init__(
    self,
    session_key: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict
):
    """Initializes HECConfig.

    Arguments:
        session_key: Splunk access token.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.
    """
    self._rest_client = rest_client.SplunkRestClient(
        session_key,
        "splunk_httpinput",
        scheme=scheme,
        host=host,
        port=port,
        **context
    )

create_input(name, stanza)

Create http data input.

Parameters:

Name Type Description Default
name str

HTTP data input name.

 required
stanza dict

Data input stanza content.

 required

Returns:

Type Description
dict

Created input.

Examples:

>>> from solnlib.hec_config import HECConfig
>>> hec = HECConfig(session_key)
>>> hec.create_input('my_hec_data_input',
                     {'index': 'main', 'sourcetype': 'hec'})
Source code in solnlib/hec_config.py 
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
@retry(exceptions=[binding.HTTPError])
def create_input(self, name: str, stanza: dict) -> dict:
    """Create http data input.

    Arguments:
        name: HTTP data input name.
        stanza: Data input stanza content.

    Returns:
        Created input.

    Examples:
       >>> from solnlib.hec_config import HECConfig
       >>> hec = HECConfig(session_key)
       >>> hec.create_input('my_hec_data_input',
                            {'index': 'main', 'sourcetype': 'hec'})
    """

    res = self._rest_client.inputs.create(name, self.input_type, **stanza)
    return res.content

delete_input(name)

Delete http data input.

Parameters:

Name Type Description Default
name str

HTTP data input name.

 required
Source code in solnlib/hec_config.py 
127
128
129
130
131
132
133
134
135
136
137
138
@retry(exceptions=[binding.HTTPError])
def delete_input(self, name: str):
    """Delete http data input.

    Arguments:
        name: HTTP data input name.
    """

    try:
        self._rest_client.inputs.delete(name, self.input_type)
    except KeyError:
        pass

get_input(name)

Get http data input.

Parameters:

Name Type Description Default
name str

HTTP event collector data input name.

 required

Returns:

Type Description
dict

HTTP event collector data input config dict.
Source code in solnlib/hec_config.py 
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
@retry(exceptions=[binding.HTTPError])
def get_input(self, name: str) -> dict:
    """Get http data input.

    Arguments:
        name: HTTP event collector data input name.

    Returns:
        HTTP event collector data input config dict.
    """

    res = self._do_get_input(name)
    if res:
        return res.content
    else:
        return None

get_limits()

Get HTTP input limits.

Returns:

Type Description
dict

HTTP input limits.
Source code in solnlib/hec_config.py 
163
164
165
166
167
168
169
170
171
@retry(exceptions=[binding.HTTPError])
def get_limits(self) -> dict:
    """Get HTTP input limits.

    Returns:
        HTTP input limits.
    """

    return self._rest_client.confs["limits"]["http_input"].content

get_settings()

Get http data input global settings.

Returns:

Type Description
dict

HTTP global settings, for example:

{ ‘enableSSL’: 1, ‘disabled’: 0, ‘useDeploymentServer’: 0, ‘port’: 8088 }
Source code in solnlib/hec_config.py 
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
@retry(exceptions=[binding.HTTPError])
def get_settings(self) -> dict:
    """Get http data input global settings.

    Returns:
        HTTP global settings, for example:

            {
                'enableSSL': 1,
                'disabled': 0,
                'useDeploymentServer': 0,
                'port': 8088
            }
    """

    return self._do_get_input(self.input_type).content

set_limits(limits)

Set HTTP input limits.

Parameters:

Name Type Description Default
limits dict

HTTP input limits.

 required
Source code in solnlib/hec_config.py 
173
174
175
176
177
178
179
180
181
182
@retry(exceptions=[binding.HTTPError])
def set_limits(self, limits: dict):
    """Set HTTP input limits.

    Arguments:
        limits: HTTP input limits.
    """

    res = self._rest_client.confs["limits"]["http_input"]
    res.submit(limits)

update_input(name, stanza)

Update http data input.

It will create if the data input doesn’t exist.

Parameters:

Name Type Description Default
name str

HTTP data input name.

 required
stanza dict

Data input stanza.

 required

Examples:

>>> from solnlib import HEConfig
>>> hec = HECConfig(session_key)
>>> hec.update_input('my_hec_data_input',
                     {'index': 'main', 'sourcetype': 'hec2'})
Source code in solnlib/hec_config.py 
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
@retry(exceptions=[binding.HTTPError])
def update_input(self, name: str, stanza: dict):
    """Update http data input.

    It will create if the data input doesn't exist.

    Arguments:
        name: HTTP data input name.
        stanza: Data input stanza.

    Examples:
       >>> from solnlib import HEConfig
       >>> hec = HECConfig(session_key)
       >>> hec.update_input('my_hec_data_input',
                            {'index': 'main', 'sourcetype': 'hec2'})
    """

    res = self._do_get_input(name)
    if res is None:
        return self.create_input(name, stanza)
    res.update(**stanza)

update_settings(settings)

Update http data input global settings.

Parameters:

Name Type Description Default
settings dict

HTTP global settings.

 required
Source code in solnlib/hec_config.py 
73
74
75
76
77
78
79
80
81
82
@retry(exceptions=[binding.HTTPError])
def update_settings(self, settings: dict):
    """Update http data input global settings.

    Arguments:
        settings: HTTP global settings.
    """

    res = self._do_get_input(self.input_type)
    res.update(**settings)

log.py

This module provides log functionalities.

__all__ = ['log_enter_exit', 'LogException', 'Logs'] module-attribute

log_authentication_error = partial(_base_error_log, exe_label='Authentication Error') module-attribute

log_configuration_error = partial(_base_error_log, exe_label='Configuration Error') module-attribute

log_connection_error = partial(_base_error_log, exe_label='Connection Error') module-attribute

log_permission_error = partial(_base_error_log, exe_label='Permission Error') module-attribute

log_server_error = partial(_base_error_log, exe_label='Server Error') module-attribute

LogException

Bases: Exception

Exception raised by Logs class.

Source code in solnlib/log.py 
60
61
62
63
class LogException(Exception):
    """Exception raised by Logs class."""

    pass

Logs

A singleton class that manage all kinds of logger.

Examples:

>>> from solnlib import log
>>> log.Logs.set_context(directory='/var/log/test',
                         namespace='test')
>>> logger = log.Logs().get_logger('mymodule')
>>> logger.set_level(logging.DEBUG)
>>> logger.debug('a debug log')
Source code in solnlib/log.py 
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
class Logs(metaclass=Singleton):
    """A singleton class that manage all kinds of logger.

    Examples:
      >>> from solnlib import log
      >>> log.Logs.set_context(directory='/var/log/test',
                               namespace='test')
      >>> logger = log.Logs().get_logger('mymodule')
      >>> logger.set_level(logging.DEBUG)
      >>> logger.debug('a debug log')
    """

    # Normal logger settings
    _default_directory = None
    _default_namespace = None
    _default_log_format = (
        "%(asctime)s log_level=%(levelname)s pid=%(process)d tid=%(threadName)s "
        "file=%(filename)s:%(funcName)s:%(lineno)d | %(message)s"
    )
    _default_log_level = logging.INFO
    _default_max_bytes = 25000000
    _default_backup_count = 5

    # Default root logger settings
    _default_root_logger_log_file = "solnlib"

    @classmethod
    def set_context(cls, **context: dict):
        """Set log context.

        List of keyword arguments:

            directory: Log directory, default is splunk log root directory.
            namespace: Logger namespace, default is None.
            log_format: Log format, default is `_default_log_format`.
            log_level: Log level, default is logging.INFO.
            max_bytes: The maximum log file size before rollover, default is 25000000.
            backup_count: The number of log files to retain,default is 5.
            root_logger_log_file: Root logger log file name, default is 'solnlib'   .

        Arguments:
            context: Keyword arguments. See list of arguments above.
        """
        if "directory" in context:
            cls._default_directory = context["directory"]
        if "namespace" in context:
            cls._default_namespace = context["namespace"]
        if "log_format" in context:
            cls._default_log_format = context["log_format"]
        if "log_level" in context:
            cls._default_log_level = context["log_level"]
        if "max_bytes" in context:
            cls._default_max_bytes = context["max_bytes"]
        if "backup_count" in context:
            cls._default_backup_count = context["backup_count"]
        if "root_logger_log_file" in context:
            cls._default_root_logger_log_file = context["root_logger_log_file"]
            cls._reset_root_logger()

    @classmethod
    def _reset_root_logger(cls):
        logger = logging.getLogger()
        log_file = cls._get_log_file(cls._default_root_logger_log_file)
        file_handler = logging.handlers.RotatingFileHandler(
            log_file,
            mode="a",
            maxBytes=cls._default_max_bytes,
            backupCount=cls._default_backup_count,
        )
        file_handler.setFormatter(logging.Formatter(cls._default_log_format))
        logger.addHandler(file_handler)
        logger.setLevel(cls._default_log_level)

    @classmethod
    def _get_log_file(cls, name):
        if cls._default_namespace:
            name = f"{cls._default_namespace}_{name}.log"
        else:
            name = f"{name}.log"

        if cls._default_directory:
            directory = cls._default_directory
        else:
            try:
                directory = make_splunkhome_path(["var", "log", "splunk"])
            except KeyError:
                raise LogException(
                    "Log directory is empty, please set log directory "
                    'by calling Logs.set_context(directory="/var/log/...").'
                )
        log_file = op.sep.join([directory, name])

        return log_file

    def __init__(self):
        self._lock = Lock()
        self._loggers = {}

    def get_logger(self, name: str) -> logging.Logger:
        """Get logger with the name of `name`.

        If logger with the name of `name` exists just return else create a new
        logger with the name of `name`.

        Arguments:
            name: Logger name, it will be used as log file name too.

        Returns:
            A named logger.
        """

        with self._lock:
            log_file = self._get_log_file(name)
            if log_file in self._loggers:
                return self._loggers[log_file]

            logger = logging.getLogger(log_file)
            handler_exists = any(
                [True for h in logger.handlers if h.baseFilename == log_file]
            )
            if not handler_exists:
                file_handler = logging.handlers.RotatingFileHandler(
                    log_file,
                    mode="a",
                    maxBytes=self._default_max_bytes,
                    backupCount=self._default_backup_count,
                )
                file_handler.setFormatter(logging.Formatter(self._default_log_format))
                logger.addHandler(file_handler)
                logger.setLevel(self._default_log_level)
                logger.propagate = False

            self._loggers[log_file] = logger
            return logger

    def set_level(self, level: int, name: str = None):
        """Set log level of logger.

        Set log level of all logger if `name` is None else of
        logger with the name of `name`.

        Arguments:
            level: Log level to set.
            name: The name of logger, default is None.
        """

        with self._lock:
            if name:
                log_file = self._get_log_file(name)
                logger = self._loggers.get(log_file)
                if logger:
                    logger.setLevel(level)
            else:
                self._default_log_level = level
                for logger in list(self._loggers.values()):
                    logger.setLevel(level)
                logging.getLogger().setLevel(level)

__init__()

Source code in solnlib/log.py 
160
161
162
def __init__(self):
    self._lock = Lock()
    self._loggers = {}

get_logger(name)

Get logger with the name of name.

If logger with the name of name exists just return else create a new logger with the name of name.

Parameters:

Name Type Description Default
name str

Logger name, it will be used as log file name too.

 required

Returns:

Type Description
logging.Logger

A named logger.
Source code in solnlib/log.py 
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
def get_logger(self, name: str) -> logging.Logger:
    """Get logger with the name of `name`.

    If logger with the name of `name` exists just return else create a new
    logger with the name of `name`.

    Arguments:
        name: Logger name, it will be used as log file name too.

    Returns:
        A named logger.
    """

    with self._lock:
        log_file = self._get_log_file(name)
        if log_file in self._loggers:
            return self._loggers[log_file]

        logger = logging.getLogger(log_file)
        handler_exists = any(
            [True for h in logger.handlers if h.baseFilename == log_file]
        )
        if not handler_exists:
            file_handler = logging.handlers.RotatingFileHandler(
                log_file,
                mode="a",
                maxBytes=self._default_max_bytes,
                backupCount=self._default_backup_count,
            )
            file_handler.setFormatter(logging.Formatter(self._default_log_format))
            logger.addHandler(file_handler)
            logger.setLevel(self._default_log_level)
            logger.propagate = False

        self._loggers[log_file] = logger
        return logger

set_context(**context) classmethod

Set log context.

List of keyword arguments

directory: Log directory, default is splunk log root directory. namespace: Logger namespace, default is None. log_format: Log format, default is _default_log_format. log_level: Log level, default is logging.INFO. max_bytes: The maximum log file size before rollover, default is 25000000. backup_count: The number of log files to retain,default is 5. root_logger_log_file: Root logger log file name, default is ‘solnlib’ .

Parameters:

Name Type Description Default
context dict

Keyword arguments. See list of arguments above.

 {}
Source code in solnlib/log.py 
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
@classmethod
def set_context(cls, **context: dict):
    """Set log context.

    List of keyword arguments:

        directory: Log directory, default is splunk log root directory.
        namespace: Logger namespace, default is None.
        log_format: Log format, default is `_default_log_format`.
        log_level: Log level, default is logging.INFO.
        max_bytes: The maximum log file size before rollover, default is 25000000.
        backup_count: The number of log files to retain,default is 5.
        root_logger_log_file: Root logger log file name, default is 'solnlib'   .

    Arguments:
        context: Keyword arguments. See list of arguments above.
    """
    if "directory" in context:
        cls._default_directory = context["directory"]
    if "namespace" in context:
        cls._default_namespace = context["namespace"]
    if "log_format" in context:
        cls._default_log_format = context["log_format"]
    if "log_level" in context:
        cls._default_log_level = context["log_level"]
    if "max_bytes" in context:
        cls._default_max_bytes = context["max_bytes"]
    if "backup_count" in context:
        cls._default_backup_count = context["backup_count"]
    if "root_logger_log_file" in context:
        cls._default_root_logger_log_file = context["root_logger_log_file"]
        cls._reset_root_logger()

set_level(level, name=None)

Set log level of logger.

Set log level of all logger if name is None else of logger with the name of name.

Parameters:

Name Type Description Default
level int

Log level to set.

 required
name str

The name of logger, default is None.

 None
Source code in solnlib/log.py 
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
def set_level(self, level: int, name: str = None):
    """Set log level of logger.

    Set log level of all logger if `name` is None else of
    logger with the name of `name`.

    Arguments:
        level: Log level to set.
        name: The name of logger, default is None.
    """

    with self._lock:
        if name:
            log_file = self._get_log_file(name)
            logger = self._loggers.get(log_file)
            if logger:
                logger.setLevel(level)
        else:
            self._default_log_level = level
            for logger in list(self._loggers.values()):
                logger.setLevel(level)
            logging.getLogger().setLevel(level)

events_ingested(logger, modular_input_name, sourcetype, n_events, index, account=None, host=None, license_usage_source=None)

Specific function to log the basic information of events ingested for the monitoring dashboard.

Parameters:

Name Type Description Default
logger logging.Logger

Add-on logger.

 required
modular_input_name str

Full name of the modular input. It needs to be in a format <input_type>://<input_name>. In case of invalid format ValueError is raised.

 required
sourcetype str

Source type used to write event.

 required
n_events int

Number of ingested events.

 required
index str

Index used to write event.

 required
license_usage_source str

source used to match data with license_usage.log.

 None
account str

Account used to write event. (optional)

 None
host str

Host used to write event. (optional)

 None
Source code in solnlib/log.py 
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
def events_ingested(
    logger: logging.Logger,
    modular_input_name: str,
    sourcetype: str,
    n_events: int,
    index: str,
    account: str = None,
    host: str = None,
    license_usage_source: str = None,
):
    """Specific function to log the basic information of events ingested for
    the monitoring dashboard.

    Arguments:
        logger: Add-on logger.
        modular_input_name: Full name of the modular input. It needs to be in a format `<input_type>://<input_name>`.
            In case of invalid format ValueError is raised.
        sourcetype: Source type used to write event.
        n_events: Number of ingested events.
        index: Index used to write event.
        license_usage_source: source used to match data with license_usage.log.
        account: Account used to write event. (optional)
        host: Host used to write event. (optional)
    """

    if "://" in modular_input_name:
        input_name = modular_input_name.split("/")[-1]
    else:
        raise ValueError(
            f"Invalid modular input name: {modular_input_name}. "
            f"It should be in format <input_type>://<input_name>"
        )

    result = {
        "action": "events_ingested",
        "modular_input_name": license_usage_source
        if license_usage_source
        else modular_input_name,
        "sourcetype_ingested": sourcetype,
        "n_events": n_events,
        "event_input": input_name,
        "event_index": index,
    }

    if account:
        result["event_account"] = account

    if host:
        result["event_host"] = host

    log_event(logger, result)

log_enter_exit(logger)

Decorator for logger to log function enter and exit.

This decorator will generate a lot of debug log, please add this only when it is required.

Parameters:

Name Type Description Default
logger logging.Logger

Logger to decorate.

 required

Examples:

>>> @log_enter_exit
>>> def myfunc():
>>>     doSomething()
Source code in solnlib/log.py 
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
def log_enter_exit(logger: logging.Logger):
    """Decorator for logger to log function enter and exit.

    This decorator will generate a lot of debug log, please add this
    only when it is required.

    Arguments:
        logger: Logger to decorate.

    Examples:
        >>> @log_enter_exit
        >>> def myfunc():
        >>>     doSomething()
    """

    def log_decorator(func):
        def wrapper(*args, **kwargs):
            logger.debug("%s entered", func.__name__)
            result = func(*args, **kwargs)
            logger.debug("%s exited", func.__name__)
            return result

        return wrapper

    return log_decorator

log_event(logger, key_values, log_level=logging.INFO)

General function to log any event in key-value format.

Source code in solnlib/log.py 
225
226
227
228
229
230
def log_event(
    logger: logging.Logger, key_values: Dict[str, Any], log_level: int = logging.INFO
):
    """General function to log any event in key-value format."""
    message = " ".join([f"{k}={v}" for k, v in key_values.items()])
    logger.log(log_level, message)

log_exception(logger, e, exc_label, full_msg=True, msg_before=None, msg_after=None, log_level=logging.ERROR)

General function to log exceptions.

Parameters:

Name Type Description Default
logger logging.Logger

Add-on logger.

 required
e Exception

Exception to log.

 required
exc_label str

label for the error to categorize it.

 required
full_msg bool

if set to True, full traceback will be logged. Default: True

 True
msg_before str

custom message before exception traceback. Default: None

 None
msg_after str

custom message after exception traceback. Default: None

 None
log_level int

Log level to log exception. Default: ERROR.

 logging.ERROR
Source code in solnlib/log.py 
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
def log_exception(
    logger: logging.Logger,
    e: Exception,
    exc_label: str,
    full_msg: bool = True,
    msg_before: str = None,
    msg_after: str = None,
    log_level: int = logging.ERROR,
):
    """General function to log exceptions.

    Arguments:
        logger: Add-on logger.
        e: Exception to log.
        exc_label: label for the error to categorize it.
        full_msg: if set to True, full traceback will be logged. Default: True
        msg_before: custom message before exception traceback. Default: None
        msg_after: custom message after exception traceback. Default: None
        log_level: Log level to log exception. Default: ERROR.
    """

    msg = _get_exception_message(e, full_msg, msg_before, msg_after)
    logger.log(log_level, f'exc_l="{exc_label}" {msg}')

modular_input_end(logger, modular_input_name)

Specific function to log the end of the modular input.

Source code in solnlib/log.py 
244
245
246
247
248
249
250
251
252
def modular_input_end(logger: logging.Logger, modular_input_name: str):
    """Specific function to log the end of the modular input."""
    log_event(
        logger,
        {
            "action": "ended",
            "modular_input_name": modular_input_name,
        },
    )

modular_input_start(logger, modular_input_name)

Specific function to log the start of the modular input.

Source code in solnlib/log.py 
233
234
235
236
237
238
239
240
241
def modular_input_start(logger: logging.Logger, modular_input_name: str):
    """Specific function to log the start of the modular input."""
    log_event(
        logger,
        {
            "action": "started",
            "modular_input_name": modular_input_name,
        },
    )

net_utils.py

Net utilities.

__all__ = ['resolve_hostname', 'validate_scheme_host_port'] module-attribute

is_valid_hostname(hostname)

Validate a host name.

Parameters:

Name Type Description Default
hostname str

host name to validate.

 required

Returns:

Type Description
bool

True if is valid else False.
Source code in solnlib/net_utils.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
def is_valid_hostname(hostname: str) -> bool:
    """Validate a host name.

    Arguments:
        hostname: host name to validate.

    Returns:
        True if is valid else False.
    """
    # Splunk IPv6 support.
    # https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/ConfigureSplunkforIPv6#Change_the_prioritization_of_IPv4_and_IPv6_communications
    if hostname == "[::1]":
        return True

    if len(hostname) > 255:
        return False
    if hostname[-1:] == ".":
        hostname = hostname[:-1]
    allowed = re.compile(r"(?!-)(::)?[A-Z\d-]{1,63}(?<!-)$", re.IGNORECASE)
    return all(allowed.match(x) for x in hostname.split("."))

is_valid_ip(addr)

Validate an IPV4 address.

Parameters:

Name Type Description Default
addr str

IP address to validate.

 required

Returns:

Type Description
bool

True if is valid else False.
Source code in solnlib/net_utils.py 
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
def is_valid_ip(addr: str) -> bool:
    """Validate an IPV4 address.

    Arguments:
        addr: IP address to validate.

    Returns:
        True if is valid else False.
    """

    ip_rx = re.compile(
        r"""
        ^(((
              [0-1]\d{2}                  # matches 000-199
            | 2[0-4]\d                    # matches 200-249
            | 25[0-5]                     # matches 250-255
            | \d{1,2}                     # matches 0-9, 00-99
        )\.){3})                          # 3 of the preceding stanzas
        ([0-1]\d{2}|2[0-4]\d|25[0-5]|\d{1,2})$     # final octet
    """,
        re.VERBOSE,
    )

    try:
        return ip_rx.match(addr.strip())
    except AttributeError:
        # Value was not a string
        return False

is_valid_port(port)

Validate a port.

Parameters:

Name Type Description Default
port Union[str, int]

port to validate.

 required

Returns:

Type Description
bool

True if is valid else False.
Source code in solnlib/net_utils.py 
110
111
112
113
114
115
116
117
118
119
120
121
122
123
def is_valid_port(port: Union[str, int]) -> bool:
    """Validate a port.

    Arguments:
        port: port to validate.

    Returns:
        True if is valid else False.
    """

    try:
        return 0 < int(port) <= 65535
    except ValueError:
        return False

is_valid_scheme(scheme)

Validate a scheme.

Parameters:

Name Type Description Default
scheme str

scheme to validate.

 required

Returns:

Type Description
bool

True if is valid else False.
Source code in solnlib/net_utils.py 
126
127
128
129
130
131
132
133
134
135
136
def is_valid_scheme(scheme: str) -> bool:
    """Validate a scheme.

    Arguments:
        scheme: scheme to validate.

    Returns:
        True if is valid else False.
    """

    return scheme.lower() in ("http", "https")

resolve_hostname(addr)

Try to resolve an IP to a host name and returns None on common failures.

Parameters:

Name Type Description Default
addr str

IP address to resolve.

 required

Returns:

Type Description
Optional[str]

Host name if success else None.

Raises:

Type Description
ValueError

If addr is not a valid address.
Source code in solnlib/net_utils.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
def resolve_hostname(addr: str) -> Optional[str]:
    """Try to resolve an IP to a host name and returns None on common failures.

    Arguments:
        addr: IP address to resolve.

    Returns:
        Host name if success else None.

    Raises:
        ValueError: If `addr` is not a valid address.
    """

    if is_valid_ip(addr):
        try:
            name, _, _ = socket.gethostbyaddr(addr)
            return name
        except socket.gaierror:
            # [Errno 8] nodename nor servname provided, or not known
            pass
        except socket.herror:
            # [Errno 1] Unknown host
            pass
        except socket.timeout:
            # Timeout.
            pass

        return None
    else:
        raise ValueError("Invalid ip address.")

validate_scheme_host_port(scheme, host, port)

Validates scheme, host and port.

Parameters:

Name Type Description Default
scheme str

scheme to validate.

 required
host str

hostname to validate.

 required
port Union[str, int]

port to validate.

 required

Raises:

Type Description
ValueError

if scheme, host or port are invalid.
Source code in solnlib/net_utils.py 
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
def validate_scheme_host_port(scheme: str, host: str, port: Union[str, int]):
    """Validates scheme, host and port.

    Arguments:
        scheme: scheme to validate.
        host: hostname to validate.
        port: port to validate.

    Raises:
        ValueError: if scheme, host or port are invalid.
    """
    if scheme is not None and not is_valid_scheme(scheme):
        raise ValueError("Invalid scheme")
    if host is not None and not is_valid_hostname(host):
        raise ValueError("Invalid host")
    if port is not None and not is_valid_port(port):
        raise ValueError("Invalid port")

orphan_process_monitor.py

Orphan process monitor.

__all__ = ['OrphanProcessChecker', 'OrphanProcessMonitor'] module-attribute

OrphanProcessChecker

Orphan process checker.

Only work for Linux platform. On Windows platform, is_orphan is always False and there is no need to do this monitoring on Windows.

Source code in solnlib/orphan_process_monitor.py 
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
class OrphanProcessChecker:
    """Orphan process checker.

    Only work for Linux platform. On Windows platform, is_orphan is
    always False and there is no need to do this monitoring on Windows.
    """

    def __init__(self, callback: Callable = None):
        """Initializes OrphanProcessChecker.

        Arguments:
            callback: (optional) Callback for orphan process.
        """
        if os.name == "nt":
            self._ppid = 0
        else:
            self._ppid = os.getppid()
        self._callback = callback

    def is_orphan(self) -> bool:
        """Check process is orphan.

        For windows platform just return False.

        Returns:
            True for orphan process else False.
        """

        if os.name == "nt":
            return False
        return self._ppid != os.getppid()

    def check_orphan(self) -> bool:
        """Check if the process becomes orphan.

        If the process becomes orphan then call callback function
        to handle properly.

        Returns:
            True for orphan process else False.
        """

        res = self.is_orphan()
        if res and self._callback:
            self._callback()
        return res

__init__(callback=None)

Initializes OrphanProcessChecker.

Parameters:

Name Type Description Default
callback Callable

(optional) Callback for orphan process.

 None
Source code in solnlib/orphan_process_monitor.py 
34
35
36
37
38
39
40
41
42
43
44
def __init__(self, callback: Callable = None):
    """Initializes OrphanProcessChecker.

    Arguments:
        callback: (optional) Callback for orphan process.
    """
    if os.name == "nt":
        self._ppid = 0
    else:
        self._ppid = os.getppid()
    self._callback = callback

check_orphan()

Check if the process becomes orphan.

If the process becomes orphan then call callback function to handle properly.

Returns:

Type Description
bool

True for orphan process else False.
Source code in solnlib/orphan_process_monitor.py 
59
60
61
62
63
64
65
66
67
68
69
70
71
72
def check_orphan(self) -> bool:
    """Check if the process becomes orphan.

    If the process becomes orphan then call callback function
    to handle properly.

    Returns:
        True for orphan process else False.
    """

    res = self.is_orphan()
    if res and self._callback:
        self._callback()
    return res

is_orphan()

Check process is orphan.

For windows platform just return False.

Returns:

Type Description
bool

True for orphan process else False.
Source code in solnlib/orphan_process_monitor.py 
46
47
48
49
50
51
52
53
54
55
56
57
def is_orphan(self) -> bool:
    """Check process is orphan.

    For windows platform just return False.

    Returns:
        True for orphan process else False.
    """

    if os.name == "nt":
        return False
    return self._ppid != os.getppid()

OrphanProcessMonitor

Orphan process monitor.

Check if process become orphan in background thread per interval and call callback if process become orphan.

Source code in solnlib/orphan_process_monitor.py 
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
class OrphanProcessMonitor:
    """Orphan process monitor.

    Check if process become orphan in background thread per interval and
    call callback if process become orphan.
    """

    def __init__(self, callback: Callable, interval: int = 1):
        """Initializes OrphanProcessMonitor.

        Arguments:
            callback: Callback for orphan process monitor.
            interval: (optional) Interval to monitor.
        """
        self._checker = OrphanProcessChecker(callback)
        self._thr = threading.Thread(target=self._do_monitor)
        self._thr.daemon = True
        self._started = False
        self._interval = interval

    def start(self):
        """Start orphan process monitor."""

        if self._started:
            return
        self._started = True

        self._thr.start()

    def stop(self):
        """Stop orphan process monitor."""

        joinable = self._started
        self._started = False
        if joinable:
            self._thr.join(timeout=1)

    def _do_monitor(self):
        while self._started:
            if self._checker.check_orphan():
                break

            for _ in range(self._interval):
                if not self._started:
                    break
                time.sleep(1)

__init__(callback, interval=1)

Initializes OrphanProcessMonitor.

Parameters:

Name Type Description Default
callback Callable

Callback for orphan process monitor.

 required
interval int

(optional) Interval to monitor.

 1
Source code in solnlib/orphan_process_monitor.py 
82
83
84
85
86
87
88
89
90
91
92
93
def __init__(self, callback: Callable, interval: int = 1):
    """Initializes OrphanProcessMonitor.

    Arguments:
        callback: Callback for orphan process monitor.
        interval: (optional) Interval to monitor.
    """
    self._checker = OrphanProcessChecker(callback)
    self._thr = threading.Thread(target=self._do_monitor)
    self._thr.daemon = True
    self._started = False
    self._interval = interval

start()

Start orphan process monitor.

Source code in solnlib/orphan_process_monitor.py 
 95
 96
 97
 98
 99
100
101
102
def start(self):
    """Start orphan process monitor."""

    if self._started:
        return
    self._started = True

    self._thr.start()

stop()

Stop orphan process monitor.

Source code in solnlib/orphan_process_monitor.py 
104
105
106
107
108
109
110
def stop(self):
    """Stop orphan process monitor."""

    joinable = self._started
    self._started = False
    if joinable:
        self._thr.join(timeout=1)

pattern.py

This module provides some common used patterns.

__all__ = ['Singleton'] module-attribute

Singleton

Bases: type

Singleton meta class.

Examples:

>>> class Test(object):
>>>     __metaclass__ = Singleton
>>>
>>>     def __init__(self):
>>>         pass
Source code in solnlib/pattern.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
class Singleton(type):
    """Singleton meta class.

    Examples:
       >>> class Test(object):
       >>>     __metaclass__ = Singleton
       >>>
       >>>     def __init__(self):
       >>>         pass
    """

    def __init__(cls, name, bases, attrs):
        super().__init__(name, bases, attrs)
        cls._instance = None

    def __call__(cls, *args, **kwargs):
        if cls._instance is None:
            cls._instance = super().__call__(*args, **kwargs)
        return cls._instance

__call__(*args, **kwargs)

Source code in solnlib/pattern.py 
37
38
39
40
def __call__(cls, *args, **kwargs):
    if cls._instance is None:
        cls._instance = super().__call__(*args, **kwargs)
    return cls._instance

__init__(name, bases, attrs)

Source code in solnlib/pattern.py 
33
34
35
def __init__(cls, name, bases, attrs):
    super().__init__(name, bases, attrs)
    cls._instance = None

server_info.py

This module contains Splunk server info related functionalities.

__all__ = ['ServerInfo', 'ServerInfoException'] module-attribute

ServerInfo

This class is a wrapper of Splunk server info.

Source code in solnlib/server_info.py 
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
class ServerInfo:
    """This class is a wrapper of Splunk server info."""

    SHC_MEMBER_ENDPOINT = "/services/shcluster/member/members"
    SHC_CAPTAIN_INFO_ENDPOINT = "/services/shcluster/captain/info"

    def __init__(
        self,
        session_key: str,
        scheme: Optional[str] = None,
        host: Optional[str] = None,
        port: Optional[int] = None,
        **context: Any
    ):
        """Initializes ServerInfo.

        Arguments:
            session_key: Splunk access token.
            scheme: The access scheme, default is None.
            host: The host name, default is None.
            port: The port number, default is None.
            context: Other configurations for Splunk rest client.
        """
        is_localhost = False
        if not all([scheme, host, port]) and os.environ.get("SPLUNK_HOME"):
            scheme, host, port = get_splunkd_access_info()
            is_localhost = (
                host == "localhost" or host == "127.0.0.1" or host in ("::1", "[::1]")
            )

        if getWebCertFile() and getWebKeyFile():
            context["cert_file"] = getWebCertFile()
            context["key_file"] = getWebKeyFile()

            if all([is_localhost, context.get("verify") is None]):
                # NOTE: this is specifically for mTLS communication
                # ONLY if scheme, host, port aren't provided AND user hasn't provided server certificate
                # we set verify to off (similar to 'rest.simpleRequest' implementation)
                context["verify"] = False

        elif getWebCertFile() is not None:
            context["cert_file"] = getWebCertFile()
            if all([is_localhost, context.get("verify") is None]):
                context["verify"] = False

        self._rest_client = rest_client.SplunkRestClient(
            session_key, "-", scheme=scheme, host=host, port=port, **context
        )

    @classmethod
    def from_server_uri(
        cls, server_uri: str, session_key: str, **context: Any
    ) -> "ServerInfo":
        """Creates ServerInfo class using server_uri and session_key.

        Note: splunktalib uses these parameters to create it's ServerInfo class,
        so this method should ease the transition from splunktalib to solnlib.

        Arguments:
            server_uri: splunkd URI.
            session_key: Splunk access token.
            context: Other configurations for Splunk rest client.

        Returns:
            An instance of `ServerInfo`.

        Raises:
            ValueError: server_uri is in the wrong format.
        """
        scheme, host, port = utils.extract_http_scheme_host_port(server_uri)
        return ServerInfo(
            session_key,
            scheme=scheme,
            host=host,
            port=port,
            **context,
        )

    def to_dict(self) -> Dict:
        """Returns server information in a form dictionary.

        Note: This method is implemented here to have compatibility with splunktalib's
        analogue.

        Returns:
            Server information in a dictionary format.
        """
        return self._server_info()

    @utils.retry(exceptions=[binding.HTTPError])
    def _server_info(self):
        return self._rest_client.info

    @property
    def server_name(self) -> str:
        """Get server name.

        Returns:
            Server name.
        """
        return self._server_info()["serverName"]

    @property
    def guid(self) -> str:
        """Get guid for the server.

        Returns:
            Server GUID.
        """
        return self._server_info()["guid"]

    @property
    def version(self) -> str:
        """Get Splunk server version.

        Returns:
            Splunk version.
        """
        return self._server_info()["version"]

    def is_captain(self) -> bool:
        """Check if this server is SHC captain.

        Note during a rolling start of SH members, the captain may be changed
        from machine to machine. To avoid the race condition, client may need
        do necessary sleep and then poll `is_captain_ready() == True` and then
        check `is_captain()`. See `is_captain_ready()` for more details.

        Returns:
            True if this server is SHC captain else False.
        """

        return "shc_captain" in self._server_info()["server_roles"]

    def is_cloud_instance(self) -> bool:
        """Check if this server is a cloud instance.

        Returns:
            True if this server is a cloud instance else False.
        """

        try:
            return self._server_info()["instance_type"] == "cloud"
        except KeyError:
            return False

    def is_search_head(self) -> bool:
        """Check if this server is a search head.

        Returns:
            True if this server is a search head else False.
        """

        server_info = self._server_info()
        for sh in ("search_head", "cluster_search_head"):
            if sh in server_info["server_roles"]:
                return True

        return False

    def is_shc_member(self) -> bool:
        """Check if this server is a SHC member.

        Returns:
            True if this server is a SHC member else False.
        """

        server_info = self._server_info()
        for sh in ("shc_member", "shc_captain"):
            if sh in server_info["server_roles"]:
                return True

        return False

    @utils.retry(exceptions=[binding.HTTPError])
    def get_shc_members(self) -> list:
        """Get SHC members.

        Raises:
            ServerInfoException: If this server has no SHC members.

        Returns:
            List of SHC members [(label, peer_scheme_host_port) ...].
        """
        try:
            content = self._rest_client.get(
                self.SHC_MEMBER_ENDPOINT, output_mode="json"
            ).body.read()
        except binding.HTTPError as e:
            if e.status != 404 and e.status != 503:
                raise

            raise ServerInfoException(
                "This server is not a SHC member and has no SHC members."
            )

        members = []
        for member in json.loads(content)["entry"]:
            content = member["content"]
            members.append((content["label"], content["peer_scheme_host_port"]))

        return members

    @utils.retry(exceptions=[binding.HTTPError])
    def is_captain_ready(self) -> bool:
        """Check if captain is ready.

        Client usually first polls this function until captain is ready
        and then call is_captain to detect current captain machine

        Returns:
            True if captain is ready else False.

        Examples:
            >>> from solnlib import server_info
            >>> serverinfo = server_info.ServerInfo(session_key)
            >>> while 1:
            >>>    if serverinfo.is_captain_ready():
            >>>        break
            >>>    time.sleep(2)
            >>>
            >>> # If do_stuff can only be executed in SH captain
            >>> if serverinfo.is_captain():
            >>>    do_stuff()
        """

        cap_info = self.captain_info()
        return utils.is_true(cap_info["service_ready_flag"]) and utils.is_false(
            cap_info["maintenance_mode"]
        )

    @utils.retry(exceptions=[binding.HTTPError])
    def captain_info(self) -> dict:
        """Get captain information.

        Raises:
            ServerInfoException: If there is SHC is not enabled.

        Returns:
            Captain information.
        """

        try:
            content = self._rest_client.get(
                self.SHC_CAPTAIN_INFO_ENDPOINT, output_mode="json"
            ).body.read()
        except binding.HTTPError as e:
            if e.status == 503 and "not available" in str(e):
                raise ServerInfoException(str(e))
            raise

        return json.loads(content)["entry"][0]["content"]

SHC_CAPTAIN_INFO_ENDPOINT = '/services/shcluster/captain/info' instance-attribute class-attribute

SHC_MEMBER_ENDPOINT = '/services/shcluster/member/members' instance-attribute class-attribute

guid: str property

Get guid for the server.

Returns:

Type Description
str

Server GUID.

server_name: str property

Get server name.

Returns:

Type Description
str

Server name.

version: str property

Get Splunk server version.

Returns:

Type Description
str

Splunk version.

__init__(session_key, scheme=None, host=None, port=None, **context)

Initializes ServerInfo.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
scheme Optional[str]

The access scheme, default is None.

 None
host Optional[str]

The host name, default is None.

 None
port Optional[int]

The port number, default is None.

 None
context Any

Other configurations for Splunk rest client.

 {}
Source code in solnlib/server_info.py 
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
def __init__(
    self,
    session_key: str,
    scheme: Optional[str] = None,
    host: Optional[str] = None,
    port: Optional[int] = None,
    **context: Any
):
    """Initializes ServerInfo.

    Arguments:
        session_key: Splunk access token.
        scheme: The access scheme, default is None.
        host: The host name, default is None.
        port: The port number, default is None.
        context: Other configurations for Splunk rest client.
    """
    is_localhost = False
    if not all([scheme, host, port]) and os.environ.get("SPLUNK_HOME"):
        scheme, host, port = get_splunkd_access_info()
        is_localhost = (
            host == "localhost" or host == "127.0.0.1" or host in ("::1", "[::1]")
        )

    if getWebCertFile() and getWebKeyFile():
        context["cert_file"] = getWebCertFile()
        context["key_file"] = getWebKeyFile()

        if all([is_localhost, context.get("verify") is None]):
            # NOTE: this is specifically for mTLS communication
            # ONLY if scheme, host, port aren't provided AND user hasn't provided server certificate
            # we set verify to off (similar to 'rest.simpleRequest' implementation)
            context["verify"] = False

    elif getWebCertFile() is not None:
        context["cert_file"] = getWebCertFile()
        if all([is_localhost, context.get("verify") is None]):
            context["verify"] = False

    self._rest_client = rest_client.SplunkRestClient(
        session_key, "-", scheme=scheme, host=host, port=port, **context
    )

captain_info()

Get captain information.

Raises:

Type Description
ServerInfoException

If there is SHC is not enabled.

Returns:

Type Description
dict

Captain information.
Source code in solnlib/server_info.py 
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
@utils.retry(exceptions=[binding.HTTPError])
def captain_info(self) -> dict:
    """Get captain information.

    Raises:
        ServerInfoException: If there is SHC is not enabled.

    Returns:
        Captain information.
    """

    try:
        content = self._rest_client.get(
            self.SHC_CAPTAIN_INFO_ENDPOINT, output_mode="json"
        ).body.read()
    except binding.HTTPError as e:
        if e.status == 503 and "not available" in str(e):
            raise ServerInfoException(str(e))
        raise

    return json.loads(content)["entry"][0]["content"]

from_server_uri(server_uri, session_key, **context) classmethod

Creates ServerInfo class using server_uri and session_key.

Note: splunktalib uses these parameters to create it’s ServerInfo class, so this method should ease the transition from splunktalib to solnlib.

Parameters:

Name Type Description Default
server_uri str

splunkd URI.

 required
session_key str

Splunk access token.

 required
context Any

Other configurations for Splunk rest client.

 {}

Returns:

Type Description
ServerInfo

An instance of ServerInfo.

Raises:

Type Description
ValueError

server_uri is in the wrong format.
Source code in solnlib/server_info.py 
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
@classmethod
def from_server_uri(
    cls, server_uri: str, session_key: str, **context: Any
) -> "ServerInfo":
    """Creates ServerInfo class using server_uri and session_key.

    Note: splunktalib uses these parameters to create it's ServerInfo class,
    so this method should ease the transition from splunktalib to solnlib.

    Arguments:
        server_uri: splunkd URI.
        session_key: Splunk access token.
        context: Other configurations for Splunk rest client.

    Returns:
        An instance of `ServerInfo`.

    Raises:
        ValueError: server_uri is in the wrong format.
    """
    scheme, host, port = utils.extract_http_scheme_host_port(server_uri)
    return ServerInfo(
        session_key,
        scheme=scheme,
        host=host,
        port=port,
        **context,
    )

get_shc_members()

Get SHC members.

Raises:

Type Description
ServerInfoException

If this server has no SHC members.

Returns:

Type Description
list

List of SHC members [(label, peer_scheme_host_port) …].
Source code in solnlib/server_info.py 
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
@utils.retry(exceptions=[binding.HTTPError])
def get_shc_members(self) -> list:
    """Get SHC members.

    Raises:
        ServerInfoException: If this server has no SHC members.

    Returns:
        List of SHC members [(label, peer_scheme_host_port) ...].
    """
    try:
        content = self._rest_client.get(
            self.SHC_MEMBER_ENDPOINT, output_mode="json"
        ).body.read()
    except binding.HTTPError as e:
        if e.status != 404 and e.status != 503:
            raise

        raise ServerInfoException(
            "This server is not a SHC member and has no SHC members."
        )

    members = []
    for member in json.loads(content)["entry"]:
        content = member["content"]
        members.append((content["label"], content["peer_scheme_host_port"]))

    return members

is_captain()

Check if this server is SHC captain.

Note during a rolling start of SH members, the captain may be changed from machine to machine. To avoid the race condition, client may need do necessary sleep and then poll is_captain_ready() == True and then check is_captain(). See is_captain_ready() for more details.

Returns:

Type Description
bool

True if this server is SHC captain else False.
Source code in solnlib/server_info.py 
168
169
170
171
172
173
174
175
176
177
178
179
180
def is_captain(self) -> bool:
    """Check if this server is SHC captain.

    Note during a rolling start of SH members, the captain may be changed
    from machine to machine. To avoid the race condition, client may need
    do necessary sleep and then poll `is_captain_ready() == True` and then
    check `is_captain()`. See `is_captain_ready()` for more details.

    Returns:
        True if this server is SHC captain else False.
    """

    return "shc_captain" in self._server_info()["server_roles"]

is_captain_ready()

Check if captain is ready.

Client usually first polls this function until captain is ready and then call is_captain to detect current captain machine

Returns:

Type Description
bool

True if captain is ready else False.

Examples:

>>> from solnlib import server_info
>>> serverinfo = server_info.ServerInfo(session_key)
>>> while 1:
>>>    if serverinfo.is_captain_ready():
>>>        break
>>>    time.sleep(2)
>>>
>>> # If do_stuff can only be executed in SH captain
>>> if serverinfo.is_captain():
>>>    do_stuff()
Source code in solnlib/server_info.py 
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
@utils.retry(exceptions=[binding.HTTPError])
def is_captain_ready(self) -> bool:
    """Check if captain is ready.

    Client usually first polls this function until captain is ready
    and then call is_captain to detect current captain machine

    Returns:
        True if captain is ready else False.

    Examples:
        >>> from solnlib import server_info
        >>> serverinfo = server_info.ServerInfo(session_key)
        >>> while 1:
        >>>    if serverinfo.is_captain_ready():
        >>>        break
        >>>    time.sleep(2)
        >>>
        >>> # If do_stuff can only be executed in SH captain
        >>> if serverinfo.is_captain():
        >>>    do_stuff()
    """

    cap_info = self.captain_info()
    return utils.is_true(cap_info["service_ready_flag"]) and utils.is_false(
        cap_info["maintenance_mode"]
    )

is_cloud_instance()

Check if this server is a cloud instance.

Returns:

Type Description
bool

True if this server is a cloud instance else False.
Source code in solnlib/server_info.py 
182
183
184
185
186
187
188
189
190
191
192
def is_cloud_instance(self) -> bool:
    """Check if this server is a cloud instance.

    Returns:
        True if this server is a cloud instance else False.
    """

    try:
        return self._server_info()["instance_type"] == "cloud"
    except KeyError:
        return False

is_search_head()

Check if this server is a search head.

Returns:

Type Description
bool

True if this server is a search head else False.
Source code in solnlib/server_info.py 
194
195
196
197
198
199
200
201
202
203
204
205
206
def is_search_head(self) -> bool:
    """Check if this server is a search head.

    Returns:
        True if this server is a search head else False.
    """

    server_info = self._server_info()
    for sh in ("search_head", "cluster_search_head"):
        if sh in server_info["server_roles"]:
            return True

    return False

is_shc_member()

Check if this server is a SHC member.

Returns:

Type Description
bool

True if this server is a SHC member else False.
Source code in solnlib/server_info.py 
208
209
210
211
212
213
214
215
216
217
218
219
220
def is_shc_member(self) -> bool:
    """Check if this server is a SHC member.

    Returns:
        True if this server is a SHC member else False.
    """

    server_info = self._server_info()
    for sh in ("shc_member", "shc_captain"):
        if sh in server_info["server_roles"]:
            return True

    return False

to_dict()

Returns server information in a form dictionary.

Note: This method is implemented here to have compatibility with splunktalib’s analogue.

Returns:

Type Description
Dict

Server information in a dictionary format.
Source code in solnlib/server_info.py 
126
127
128
129
130
131
132
133
134
135
def to_dict(self) -> Dict:
    """Returns server information in a form dictionary.

    Note: This method is implemented here to have compatibility with splunktalib's
    analogue.

    Returns:
        Server information in a dictionary format.
    """
    return self._server_info()

ServerInfoException

Bases: Exception

Exception raised by ServerInfo class.

Source code in solnlib/server_info.py 
42
43
44
45
class ServerInfoException(Exception):
    """Exception raised by ServerInfo class."""

    pass

getWebCertFile()

Source code in solnlib/server_info.py 
27
28
def getWebCertFile():
    return None

getWebKeyFile()

Source code in solnlib/server_info.py 
30
31
def getWebKeyFile():
    return None

splunk_rest_client.py

This module proxy all REST call to splunklib SDK, it handles proxy, certs etc in this centralized location.

All clients should use SplunkRestProxy to do REST call instead of calling splunklib SDK directly in business logic code.

MAX_REQUEST_RETRIES = 5 module-attribute

__all__ = ['SplunkRestClient'] module-attribute

SplunkRestClient

Bases: client.Service

Splunk REST client.

Source code in solnlib/splunk_rest_client.py 
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
class SplunkRestClient(client.Service):
    """Splunk REST client."""

    def __init__(
        self,
        session_key: str,
        app: str,
        owner: str = "nobody",
        scheme: str = None,
        host: str = None,
        port: int = None,
        **context: dict,
    ):
        """Initializes SplunkRestClient.

        Arguments `scheme`, `host` and `port` are optional in the Splunk
        environment (when environment variable SPLUNK_HOME is set). In this
        situation `get_splunkd_access_info` will be used to set `scheme`,
        `host` and `port`. In case of using `SplunkRestClient` outside of
        Splunk environment - `scheme`, `host` and `port` should be provided.

        Arguments:
            session_key: Splunk access token.
            app: App name of namespace.
            owner: Owner of namespace, default is `nobody`.
            scheme: The access scheme, default is None.
            host: The host name, default is None.
            port: The port number, default is None.
            context: Other configurations, it can contain `proxy_hostname`,
                `proxy_port`, `proxy_username`, `proxy_password`, then proxy will
                be accounted and setup, all REST APIs to splunkd will be through
                the proxy. If `context` contains `key_file`, `cert_file`, then
                certification will be accounted and setup, all REST APIs to splunkd
                will use certification. If `context` contains `pool_connections`,
                `pool_maxsize`, then HTTP connection will be pooled.

        Raises:
            ValueError: if scheme, host or port are invalid.
        """
        # Only do splunkd URI discovery in SPLUNK env (SPLUNK_HOME is set).
        if not all([scheme, host, port]) and os.environ.get("SPLUNK_HOME"):
            scheme, host, port = get_splunkd_access_info()
        if os.environ.get("SPLUNK_HOME") is None:
            if not all([scheme, host, port]):
                raise ValueError(
                    "scheme, host, port should be provided outside of Splunk environment"
                )

        validate_scheme_host_port(scheme, host, port)
        if host == "[::1]":
            host = "::1"

        handler = _request_handler(context)
        super().__init__(
            handler=handler,
            scheme=scheme,
            host=host,
            port=port,
            token=session_key,
            app=app,
            owner=owner,
            autologin=True,
        )

__init__(session_key, app, owner='nobody', scheme=None, host=None, port=None, **context)

Initializes SplunkRestClient.

Arguments scheme, host and port are optional in the Splunk environment (when environment variable SPLUNK_HOME is set). In this situation get_splunkd_access_info will be used to set scheme, host and port. In case of using SplunkRestClient outside of Splunk environment - scheme, host and port should be provided.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner str

Owner of namespace, default is nobody.

 'nobody'
scheme str

The access scheme, default is None.

 None
host str

The host name, default is None.

 None
port int

The port number, default is None.

 None
context dict

Other configurations, it can contain proxy_hostname, proxy_port, proxy_username, proxy_password, then proxy will be accounted and setup, all REST APIs to splunkd will be through the proxy. If context contains key_file, cert_file, then certification will be accounted and setup, all REST APIs to splunkd will use certification. If context contains pool_connections, pool_maxsize, then HTTP connection will be pooled.

 {}

Raises:

Type Description
ValueError

if scheme, host or port are invalid.
Source code in solnlib/splunk_rest_client.py 
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
def __init__(
    self,
    session_key: str,
    app: str,
    owner: str = "nobody",
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
):
    """Initializes SplunkRestClient.

    Arguments `scheme`, `host` and `port` are optional in the Splunk
    environment (when environment variable SPLUNK_HOME is set). In this
    situation `get_splunkd_access_info` will be used to set `scheme`,
    `host` and `port`. In case of using `SplunkRestClient` outside of
    Splunk environment - `scheme`, `host` and `port` should be provided.

    Arguments:
        session_key: Splunk access token.
        app: App name of namespace.
        owner: Owner of namespace, default is `nobody`.
        scheme: The access scheme, default is None.
        host: The host name, default is None.
        port: The port number, default is None.
        context: Other configurations, it can contain `proxy_hostname`,
            `proxy_port`, `proxy_username`, `proxy_password`, then proxy will
            be accounted and setup, all REST APIs to splunkd will be through
            the proxy. If `context` contains `key_file`, `cert_file`, then
            certification will be accounted and setup, all REST APIs to splunkd
            will use certification. If `context` contains `pool_connections`,
            `pool_maxsize`, then HTTP connection will be pooled.

    Raises:
        ValueError: if scheme, host or port are invalid.
    """
    # Only do splunkd URI discovery in SPLUNK env (SPLUNK_HOME is set).
    if not all([scheme, host, port]) and os.environ.get("SPLUNK_HOME"):
        scheme, host, port = get_splunkd_access_info()
    if os.environ.get("SPLUNK_HOME") is None:
        if not all([scheme, host, port]):
            raise ValueError(
                "scheme, host, port should be provided outside of Splunk environment"
            )

    validate_scheme_host_port(scheme, host, port)
    if host == "[::1]":
        host = "::1"

    handler = _request_handler(context)
    super().__init__(
        handler=handler,
        scheme=scheme,
        host=host,
        port=port,
        token=session_key,
        app=app,
        owner=owner,
        autologin=True,
    )

bulletin_rest_client.py

__all__ = ['BulletinRestClient'] module-attribute

BulletinRestClient

REST client for handling Bulletin messages.

Source code in solnlib/bulletin_rest_client.py 
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
class BulletinRestClient:
    """REST client for handling Bulletin messages."""

    MESSAGES_ENDPOINT = "/services/messages"

    headers = [("Content-Type", "application/json")]

    class Severity:
        INFO = "info"
        WARNING = "warn"
        ERROR = "error"

    def __init__(
        self,
        message_name: str,
        session_key: str,
        app: str,
        **context: dict,
    ):
        """Initializes BulletinRestClient.
            When creating a new bulletin message, you must provide a name, which is a kind of ID.
            If you try to create another message with the same name (ID), the API will not add another message
            to the bulletin, but it will overwrite the existing one. Similar behaviour applies to deletion.
            To delete a message, you must indicate the name (ID) of the message.
            To provide better and easier control over bulletin messages, this client works in such a way
            that there is one instance responsible for handling one specific message.
            If you need to add another message to bulletin create another instance
            with a different 'message_name'
            e.g.
            msg_1 = BulletinRestClient("message_1", "<some session key>")
            msg_2 = BulletinRestClient("message_2", "<some session key>")

        Arguments:
            message_name: Name of the message in the Splunk's bulletin.
            session_key: Splunk access token.
            app: App name of namespace.
            context: Other configurations for Splunk rest client.
        """

        self.message_name = message_name
        self.session_key = session_key
        self.app = app

        self._rest_client = rest_client.SplunkRestClient(
            self.session_key, app=self.app, **context
        )

    def create_message(
        self,
        msg: str,
        severity: Severity = Severity.WARNING,
        capabilities: Optional[List[str]] = None,
        roles: Optional[List] = None,
    ):
        """Creates a message in the Splunk's bulletin. Calling this method
        multiple times for the same instance will overwrite existing message.

        Arguments:
            msg: The message which will be displayed in the Splunk's bulletin
            severity: Severity level of the message. It has to be one of: 'info', 'warn', 'error'.
                If wrong severity is given, ValueError will be raised.
            capabilities: One or more capabilities that users must have to view the message.
                Capability names are validated.
                This argument should be provided as a list of string/s e.g. capabilities=['one', 'two'].
                If a non-existent capability is used, HTTP 400 BAD REQUEST exception will be raised.
                If argument is not a List[str] ValueError will be raised.
            roles: One or more roles that users must have to view the message. Role names are validated.
                This argument should be provided as a list of string/s e.g. roles=['user', 'admin'].
                If a non-existent role is used, HTTP 400 BAD REQUEST exception will be raised.
                If argument is not a List[str] ValueError will be raised.
        """
        body = {
            "name": self.message_name,
            "value": msg,
            "severity": severity,
            "capability": [],
            "role": [],
        }

        if severity not in (
            self.Severity.INFO,
            self.Severity.WARNING,
            self.Severity.ERROR,
        ):
            raise ValueError(
                "Severity must be one of ("
                "'BulletinRestClient.Severity.INFO', "
                "'BulletinRestClient.Severity.WARNING', "
                "'BulletinRestClient.Severity.ERROR'"
                ")."
            )

        if capabilities:
            body["capability"] = self._validate_and_get_body_value(
                capabilities, "Capabilities must be a list of strings."
            )

        if roles:
            body["role"] = self._validate_and_get_body_value(
                roles, "Roles must be a list of strings."
            )

        self._rest_client.post(self.MESSAGES_ENDPOINT, body=body, headers=self.headers)

    def get_message(self):
        """Get specific message created by this instance."""
        endpoint = f"{self.MESSAGES_ENDPOINT}/{self.message_name}"
        response = self._rest_client.get(endpoint, output_mode="json").body.read()
        return json.loads(response)

    def get_all_messages(self):
        """Get all messages in the bulletin."""
        response = self._rest_client.get(
            self.MESSAGES_ENDPOINT, output_mode="json"
        ).body.read()
        return json.loads(response)

    def delete_message(self):
        """Delete specific message created by this instance."""
        endpoint = f"{self.MESSAGES_ENDPOINT}/{self.message_name}"
        self._rest_client.delete(endpoint)

    @staticmethod
    def _validate_and_get_body_value(arg, error_msg) -> List:
        if type(arg) is list and (all(isinstance(el, str) for el in arg)):
            return [el for el in arg]
        else:
            raise ValueError(error_msg)

MESSAGES_ENDPOINT = '/services/messages' instance-attribute class-attribute

app = app instance-attribute

headers = [('Content-Type', 'application/json')] instance-attribute class-attribute

message_name = message_name instance-attribute

session_key = session_key instance-attribute

Severity

Source code in solnlib/bulletin_rest_client.py 
31
32
33
34
class Severity:
    INFO = "info"
    WARNING = "warn"
    ERROR = "error"

ERROR = 'error' instance-attribute class-attribute

INFO = 'info' instance-attribute class-attribute

WARNING = 'warn' instance-attribute class-attribute

__init__(message_name, session_key, app, **context)

Initializes BulletinRestClient. When creating a new bulletin message, you must provide a name, which is a kind of ID. If you try to create another message with the same name (ID), the API will not add another message to the bulletin, but it will overwrite the existing one. Similar behaviour applies to deletion. To delete a message, you must indicate the name (ID) of the message. To provide better and easier control over bulletin messages, this client works in such a way that there is one instance responsible for handling one specific message. If you need to add another message to bulletin create another instance with a different ‘message_name’ e.g. msg_1 = BulletinRestClient(“message_1”, ““) msg_2 = BulletinRestClient(“message_2”, ““)

Parameters:

Name Type Description Default
message_name str

Name of the message in the Splunk’s bulletin.

 required
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
context dict

Other configurations for Splunk rest client.

 {}
Source code in solnlib/bulletin_rest_client.py 
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
def __init__(
    self,
    message_name: str,
    session_key: str,
    app: str,
    **context: dict,
):
    """Initializes BulletinRestClient.
        When creating a new bulletin message, you must provide a name, which is a kind of ID.
        If you try to create another message with the same name (ID), the API will not add another message
        to the bulletin, but it will overwrite the existing one. Similar behaviour applies to deletion.
        To delete a message, you must indicate the name (ID) of the message.
        To provide better and easier control over bulletin messages, this client works in such a way
        that there is one instance responsible for handling one specific message.
        If you need to add another message to bulletin create another instance
        with a different 'message_name'
        e.g.
        msg_1 = BulletinRestClient("message_1", "<some session key>")
        msg_2 = BulletinRestClient("message_2", "<some session key>")

    Arguments:
        message_name: Name of the message in the Splunk's bulletin.
        session_key: Splunk access token.
        app: App name of namespace.
        context: Other configurations for Splunk rest client.
    """

    self.message_name = message_name
    self.session_key = session_key
    self.app = app

    self._rest_client = rest_client.SplunkRestClient(
        self.session_key, app=self.app, **context
    )

create_message(msg, severity=Severity.WARNING, capabilities=None, roles=None)

Creates a message in the Splunk’s bulletin. Calling this method multiple times for the same instance will overwrite existing message.

Parameters:

Name Type Description Default
msg str

The message which will be displayed in the Splunk’s bulletin

 required
severity Severity

Severity level of the message. It has to be one of: ‘info’, ‘warn’, ‘error’. If wrong severity is given, ValueError will be raised.

 Severity.WARNING
capabilities Optional[List[str]]

One or more capabilities that users must have to view the message. Capability names are validated. This argument should be provided as a list of string/s e.g. capabilities=[‘one’, ‘two’]. If a non-existent capability is used, HTTP 400 BAD REQUEST exception will be raised. If argument is not a List[str] ValueError will be raised.

 None
roles Optional[List]

One or more roles that users must have to view the message. Role names are validated. This argument should be provided as a list of string/s e.g. roles=[‘user’, ‘admin’]. If a non-existent role is used, HTTP 400 BAD REQUEST exception will be raised. If argument is not a List[str] ValueError will be raised.

 None
Source code in solnlib/bulletin_rest_client.py 
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
def create_message(
    self,
    msg: str,
    severity: Severity = Severity.WARNING,
    capabilities: Optional[List[str]] = None,
    roles: Optional[List] = None,
):
    """Creates a message in the Splunk's bulletin. Calling this method
    multiple times for the same instance will overwrite existing message.

    Arguments:
        msg: The message which will be displayed in the Splunk's bulletin
        severity: Severity level of the message. It has to be one of: 'info', 'warn', 'error'.
            If wrong severity is given, ValueError will be raised.
        capabilities: One or more capabilities that users must have to view the message.
            Capability names are validated.
            This argument should be provided as a list of string/s e.g. capabilities=['one', 'two'].
            If a non-existent capability is used, HTTP 400 BAD REQUEST exception will be raised.
            If argument is not a List[str] ValueError will be raised.
        roles: One or more roles that users must have to view the message. Role names are validated.
            This argument should be provided as a list of string/s e.g. roles=['user', 'admin'].
            If a non-existent role is used, HTTP 400 BAD REQUEST exception will be raised.
            If argument is not a List[str] ValueError will be raised.
    """
    body = {
        "name": self.message_name,
        "value": msg,
        "severity": severity,
        "capability": [],
        "role": [],
    }

    if severity not in (
        self.Severity.INFO,
        self.Severity.WARNING,
        self.Severity.ERROR,
    ):
        raise ValueError(
            "Severity must be one of ("
            "'BulletinRestClient.Severity.INFO', "
            "'BulletinRestClient.Severity.WARNING', "
            "'BulletinRestClient.Severity.ERROR'"
            ")."
        )

    if capabilities:
        body["capability"] = self._validate_and_get_body_value(
            capabilities, "Capabilities must be a list of strings."
        )

    if roles:
        body["role"] = self._validate_and_get_body_value(
            roles, "Roles must be a list of strings."
        )

    self._rest_client.post(self.MESSAGES_ENDPOINT, body=body, headers=self.headers)

delete_message()

Delete specific message created by this instance.

Source code in solnlib/bulletin_rest_client.py 
141
142
143
144
def delete_message(self):
    """Delete specific message created by this instance."""
    endpoint = f"{self.MESSAGES_ENDPOINT}/{self.message_name}"
    self._rest_client.delete(endpoint)

get_all_messages()

Get all messages in the bulletin.

Source code in solnlib/bulletin_rest_client.py 
134
135
136
137
138
139
def get_all_messages(self):
    """Get all messages in the bulletin."""
    response = self._rest_client.get(
        self.MESSAGES_ENDPOINT, output_mode="json"
    ).body.read()
    return json.loads(response)

get_message()

Get specific message created by this instance.

Source code in solnlib/bulletin_rest_client.py 
128
129
130
131
132
def get_message(self):
    """Get specific message created by this instance."""
    endpoint = f"{self.MESSAGES_ENDPOINT}/{self.message_name}"
    response = self._rest_client.get(endpoint, output_mode="json").body.read()
    return json.loads(response)

splunkenv.py

Splunk platform related utilities.

ETC_LEAF = 'etc' module-attribute

__all__ = ['make_splunkhome_path', 'get_splunk_host_info', 'get_splunk_bin', 'get_splunkd_access_info', 'get_scheme_from_hec_settings', 'get_splunkd_uri', 'get_conf_key_value', 'get_conf_stanza', 'get_conf_stanzas'] module-attribute

on_shared_storage = [os.path.join(ETC_LEAF, 'apps'), os.path.join(ETC_LEAF, 'users'), os.path.join('var', 'run', 'splunk', 'dispatch'), os.path.join('var', 'run', 'splunk', 'srtemp'), os.path.join('var', 'run', 'splunk', 'rss'), os.path.join('var', 'run', 'splunk', 'scheduler'), os.path.join('var', 'run', 'splunk', 'lookup_tmp')] module-attribute

get_conf_key_value(conf_name, stanza, key)

Get value of key of stanza in conf_name.

Parameters:

Name Type Description Default
conf_name str

Config file.

 required
stanza str

Stanza name.

 required
key str

Key name.

 required

Returns:

Type Description
Union[str, List, dict]

Config value.

Raises:

Type Description
KeyError

If stanza or key doesn’t exist.
Source code in solnlib/splunkenv.py 
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
def get_conf_key_value(conf_name: str, stanza: str, key: str) -> Union[str, List, dict]:
    """Get value of `key` of `stanza` in `conf_name`.

    Arguments:
        conf_name: Config file.
        stanza: Stanza name.
        key: Key name.

    Returns:
        Config value.

    Raises:
        KeyError: If `stanza` or `key` doesn't exist.
    """

    stanzas = get_conf_stanzas(conf_name)
    return stanzas[stanza][key]

get_conf_stanza(conf_name, stanza)

Get stanza in conf_name.

Parameters:

Name Type Description Default
conf_name str

Config file.

 required
stanza str

Stanza name.

 required

Returns:

Type Description
dict

Config stanza.

Raises:

Type Description
KeyError

If stanza doesn’t exist.
Source code in solnlib/splunkenv.py 
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
def get_conf_stanza(conf_name: str, stanza: str) -> dict:
    """Get `stanza` in `conf_name`.

    Arguments:
        conf_name: Config file.
        stanza: Stanza name.

    Returns:
        Config stanza.

    Raises:
         KeyError: If stanza doesn't exist.
    """

    stanzas = get_conf_stanzas(conf_name)
    return stanzas[stanza]

get_conf_stanzas(conf_name)

Get stanzas of conf_name

Parameters:

Name Type Description Default
conf_name str

Config file.

 required

Returns:

Type Description
dict

Config stanzas.

Examples:

>>> stanzas = get_conf_stanzas('server')
>>> return: {'serverName': 'testServer', 'sessionTimeout': '1h', ...}
Source code in solnlib/splunkenv.py 
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
def get_conf_stanzas(conf_name: str) -> dict:
    """Get stanzas of `conf_name`

    Arguments:
        conf_name: Config file.

    Returns:
        Config stanzas.

    Examples:
       >>> stanzas = get_conf_stanzas('server')
       >>> return: {'serverName': 'testServer', 'sessionTimeout': '1h', ...}
    """

    if conf_name.endswith(".conf"):
        conf_name = conf_name[:-5]

    # TODO: dynamically calculate SPLUNK_HOME
    btool_cli = [
        op.join(os.environ["SPLUNK_HOME"], "bin", "splunk"),
        "cmd",
        "btool",
        conf_name,
        "list",
    ]
    p = subprocess.Popen(  # nosemgrep: python.lang.security.audit.dangerous-subprocess-use.dangerous-subprocess-use
        btool_cli, stdout=subprocess.PIPE, stderr=subprocess.PIPE
    )
    out, _ = p.communicate()

    if isinstance(out, bytes):
        out = out.decode()

    parser = ConfigParser(**{"strict": False})
    parser.optionxform = str
    parser.read_file(StringIO(out))

    out = {}
    for section in parser.sections():
        out[section] = {item[0]: item[1] for item in parser.items(section, raw=True)}
    return out

get_scheme_from_hec_settings()

Get scheme from HEC global settings.

Returns:

Type Description
str

scheme (str)
Source code in solnlib/splunkenv.py 
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
def get_scheme_from_hec_settings() -> str:
    """Get scheme from HEC global settings.

    Returns:
        scheme (str)
    """
    try:
        ssl_enabled = get_conf_key_value("inputs", "http", "enableSSL")
    except KeyError:
        raise KeyError(
            "Cannot get enableSSL setting form conf: 'inputs' and stanza: '[http]'. "
            "Verify that your Splunk instance has the inputs.conf file with the correct [http] stanza. "
            "For more information see: "
            "https://docs.splunk.com/Documentation/Splunk/9.2.0/Data/UseHECusingconffiles"
        )

    if is_true(ssl_enabled):
        scheme = "https"
    else:
        scheme = "http"

    return scheme

get_splunk_bin()

Get absolute path of splunk CLI.

Returns:

Type Description
str

Absolute path of splunk CLI.
Source code in solnlib/splunkenv.py 
162
163
164
165
166
167
168
169
170
171
172
173
def get_splunk_bin() -> str:
    """Get absolute path of splunk CLI.

    Returns:
        Absolute path of splunk CLI.
    """

    if os.name == "nt":
        splunk_bin = "splunk.exe"
    else:
        splunk_bin = "splunk"
    return make_splunkhome_path(("bin", splunk_bin))

get_splunk_host_info()

Get splunk host info.

Returns:

Type Description
Tuple

Tuple of (server_name, host_name).
Source code in solnlib/splunkenv.py 
150
151
152
153
154
155
156
157
158
159
def get_splunk_host_info() -> Tuple:
    """Get splunk host info.

    Returns:
        Tuple of (server_name, host_name).
    """

    server_name = get_conf_key_value("server", "general", "serverName")
    host_name = socket.gethostname()
    return server_name, host_name

get_splunkd_access_info()

Get splunkd server access info.

Returns:

Type Description
Tuple[str, str, int]

Tuple of (scheme, host, port).
Source code in solnlib/splunkenv.py 
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
def get_splunkd_access_info() -> Tuple[str, str, int]:
    """Get splunkd server access info.

    Returns:
        Tuple of (scheme, host, port).
    """

    if is_true(get_conf_key_value("server", "sslConfig", "enableSplunkdSSL")):
        scheme = "https"
    else:
        scheme = "http"

    host_port = get_conf_key_value("web", "settings", "mgmtHostPort")
    host_port = host_port.strip()
    host_port_split_parts = host_port.split(":")
    host = ":".join(host_port_split_parts[:-1])
    port = int(host_port_split_parts[-1])

    if "SPLUNK_BINDIP" in os.environ:
        bindip = os.environ["SPLUNK_BINDIP"]
        port_idx = bindip.rfind(":")
        host = bindip[:port_idx] if port_idx > 0 else bindip

    return scheme, host, port

get_splunkd_uri()

Get splunkd uri.

Returns:

Type Description
str

Splunkd uri.
Source code in solnlib/splunkenv.py 
226
227
228
229
230
231
232
233
234
235
236
237
def get_splunkd_uri() -> str:
    """Get splunkd uri.

    Returns:
        Splunkd uri.
    """

    if os.environ.get("SPLUNKD_URI"):
        return os.environ["SPLUNKD_URI"]

    scheme, host, port = get_splunkd_access_info()
    return f"{scheme}://{host}:{port}"

make_splunkhome_path(parts)

Construct absolute path by $SPLUNK_HOME and parts.

Concatenate $SPLUNK_HOME and parts to an absolute path. For example, parts is [‘etc’, ‘apps’, ‘Splunk_TA_test’], the return path will be $SPLUNK_HOME/etc/apps/Splunk_TA_test. Note: this function assumed SPLUNK_HOME is in environment varialbes.

Parameters:

Name Type Description Default
parts Union[List, Tuple]

Path parts.

 required

Returns:

Type Description
str

Absolute path.

Raises:

Type Description
ValueError

Escape from intended parent directories.
Source code in solnlib/splunkenv.py 
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
def make_splunkhome_path(parts: Union[List, Tuple]) -> str:
    """Construct absolute path by $SPLUNK_HOME and `parts`.

    Concatenate $SPLUNK_HOME and `parts` to an absolute path.
    For example, `parts` is ['etc', 'apps', 'Splunk_TA_test'],
    the return path will be $SPLUNK_HOME/etc/apps/Splunk_TA_test.
    Note: this function assumed SPLUNK_HOME is in environment varialbes.

    Arguments:
        parts: Path parts.

    Returns:
        Absolute path.

    Raises:
        ValueError: Escape from intended parent directories.
    """

    relpath = os.path.normpath(os.path.join(*parts))

    basepath = None
    shared_storage = _get_shared_storage()
    if shared_storage:
        for candidate in on_shared_storage:
            # SPL-100508 On windows if the path is missing the drive letter,
            # construct fullpath manually and call relpath
            if os.name == "nt" and not _verify_path_prefix(relpath, candidate):
                break

            if os.path.relpath(relpath, candidate)[0:2] != "..":
                basepath = shared_storage
                break

    if basepath is None:
        etc_with_trailing_sep = os.path.join(ETC_LEAF, "")
        if relpath == ETC_LEAF or relpath.startswith(etc_with_trailing_sep):
            # Redirect $SPLUNK_HOME/etc to $SPLUNK_ETC.
            basepath = _splunk_etc()
            # Remove leading etc (and path separator, if present). Note: when
            # emitting $SPLUNK_ETC exactly, with no additional path parts, we
            # set <relpath> to the empty string.
            relpath = relpath[4:]
        else:
            basepath = _splunk_home()

    fullpath = os.path.normpath(os.path.join(basepath, relpath))

    # Check that we haven't escaped from intended parent directories.
    if os.path.relpath(fullpath, basepath)[0:2] == "..":
        raise ValueError(
            f'Illegal escape from parent directory "{basepath}": {fullpath}'
        )
    return fullpath

time_parser.py

This module provides interfaces to parse and convert timestamp.

__all__ = ['TimeParser'] module-attribute

InvalidTimeFormatException

Bases: Exception

Exception for invalid time format.

Source code in solnlib/time_parser.py 
31
32
33
34
class InvalidTimeFormatException(Exception):
    """Exception for invalid time format."""

    pass

TimeParser

Datetime parser.

Use splunkd rest to parse datetime.

Examples:

>>> from solnlib import time_parser
>>> tp = time_parser.TimeParser(session_key)
>>> tp.to_seconds('2011-07-06T21:54:23.000-07:00')
>>> tp.to_utc('2011-07-06T21:54:23.000-07:00')
>>> tp.to_local('2011-07-06T21:54:23.000-07:00')
Source code in solnlib/time_parser.py 
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
class TimeParser:
    """Datetime parser.

    Use splunkd rest to parse datetime.

    Examples:
       >>> from solnlib import time_parser
       >>> tp = time_parser.TimeParser(session_key)
       >>> tp.to_seconds('2011-07-06T21:54:23.000-07:00')
       >>> tp.to_utc('2011-07-06T21:54:23.000-07:00')
       >>> tp.to_local('2011-07-06T21:54:23.000-07:00')
    """

    URL = "/services/search/timeparser"

    def __init__(
        self,
        session_key: str,
        scheme: str = None,
        host: str = None,
        port: int = None,
        **context: Any,
    ):
        """Initializes TimeParser.

        Arguments:
            session_key: Splunk access token.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            context: Other configurations for Splunk rest client.

        Raises:
            ValueError: if scheme, host or port are invalid.
        """
        self._rest_client = rest_client.SplunkRestClient(
            session_key, "-", scheme=scheme, host=host, port=port, **context
        )

    @retry(exceptions=[binding.HTTPError])
    def to_seconds(self, time_str: str) -> float:
        """Parse `time_str` and convert to seconds since epoch.

        Arguments:
            time_str: ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

        Raises:
            binding.HTTPError: rest client returns an exception (everything
                else than 400 code).
            InvalidTimeFormatException: when time format is invalid (rest
                client returns 400 code).

        Returns:
            Seconds since epoch.
        """

        try:
            response = self._rest_client.get(
                self.URL, output_mode="json", time=time_str, output_time_format="%s"
            ).body.read()
        except binding.HTTPError as e:
            if e.status != 400:
                raise

            raise InvalidTimeFormatException(f"Invalid time format: {time_str}.")

        seconds = json.loads(response)[time_str]
        return float(seconds)

    def to_utc(self, time_str: str) -> datetime.datetime:
        """Parse `time_str` and convert to UTC timestamp.

        Arguments:
            time_str: ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

        Raises:
            binding.HTTPError: rest client returns an exception (everything
                else than 400 code).
            InvalidTimeFormatException: when time format is invalid (rest
                client returns 400 code).

        Returns:
            UTC timestamp.
        """

        return datetime.datetime.utcfromtimestamp(self.to_seconds(time_str))

    @retry(exceptions=[binding.HTTPError])
    def to_local(self, time_str: str) -> str:
        """Parse `time_str` and convert to local timestamp.

        Arguments:
            time_str: ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

        Raises:
            binding.HTTPError: rest client returns an exception (everything
                else than 400 code).
            InvalidTimeFormatException: when time format is invalid (rest
                client returns 400 code).

        Returns:
            Local timestamp in ISO8601 format.
        """

        try:
            response = self._rest_client.get(
                self.URL, output_mode="json", time=time_str
            ).body.read()
        except binding.HTTPError as e:
            if e.status != 400:
                raise

            raise InvalidTimeFormatException(f"Invalid time format: {time_str}.")

        return json.loads(response)[time_str]

URL = '/services/search/timeparser' instance-attribute class-attribute

__init__(session_key, scheme=None, host=None, port=None, **context)

Initializes TimeParser.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context Any

Other configurations for Splunk rest client.

 {}

Raises:

Type Description
ValueError

if scheme, host or port are invalid.
Source code in solnlib/time_parser.py 
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
def __init__(
    self,
    session_key: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: Any,
):
    """Initializes TimeParser.

    Arguments:
        session_key: Splunk access token.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Raises:
        ValueError: if scheme, host or port are invalid.
    """
    self._rest_client = rest_client.SplunkRestClient(
        session_key, "-", scheme=scheme, host=host, port=port, **context
    )

to_local(time_str)

Parse time_str and convert to local timestamp.

Parameters:

Name Type Description Default
time_str str

ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

 required

Raises:

Type Description
binding.HTTPError

rest client returns an exception (everything else than 400 code).
InvalidTimeFormatException

when time format is invalid (rest client returns 400 code).

Returns:

Type Description
str

Local timestamp in ISO8601 format.
Source code in solnlib/time_parser.py 
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
@retry(exceptions=[binding.HTTPError])
def to_local(self, time_str: str) -> str:
    """Parse `time_str` and convert to local timestamp.

    Arguments:
        time_str: ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

    Raises:
        binding.HTTPError: rest client returns an exception (everything
            else than 400 code).
        InvalidTimeFormatException: when time format is invalid (rest
            client returns 400 code).

    Returns:
        Local timestamp in ISO8601 format.
    """

    try:
        response = self._rest_client.get(
            self.URL, output_mode="json", time=time_str
        ).body.read()
    except binding.HTTPError as e:
        if e.status != 400:
            raise

        raise InvalidTimeFormatException(f"Invalid time format: {time_str}.")

    return json.loads(response)[time_str]

to_seconds(time_str)

Parse time_str and convert to seconds since epoch.

Parameters:

Name Type Description Default
time_str str

ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

 required

Raises:

Type Description
binding.HTTPError

rest client returns an exception (everything else than 400 code).
InvalidTimeFormatException

when time format is invalid (rest client returns 400 code).

Returns:

Type Description
float

Seconds since epoch.
Source code in solnlib/time_parser.py 
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
@retry(exceptions=[binding.HTTPError])
def to_seconds(self, time_str: str) -> float:
    """Parse `time_str` and convert to seconds since epoch.

    Arguments:
        time_str: ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

    Raises:
        binding.HTTPError: rest client returns an exception (everything
            else than 400 code).
        InvalidTimeFormatException: when time format is invalid (rest
            client returns 400 code).

    Returns:
        Seconds since epoch.
    """

    try:
        response = self._rest_client.get(
            self.URL, output_mode="json", time=time_str, output_time_format="%s"
        ).body.read()
    except binding.HTTPError as e:
        if e.status != 400:
            raise

        raise InvalidTimeFormatException(f"Invalid time format: {time_str}.")

    seconds = json.loads(response)[time_str]
    return float(seconds)

to_utc(time_str)

Parse time_str and convert to UTC timestamp.

Parameters:

Name Type Description Default
time_str str

ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

 required

Raises:

Type Description
binding.HTTPError

rest client returns an exception (everything else than 400 code).
InvalidTimeFormatException

when time format is invalid (rest client returns 400 code).

Returns:

Type Description
datetime.datetime

UTC timestamp.
Source code in solnlib/time_parser.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def to_utc(self, time_str: str) -> datetime.datetime:
    """Parse `time_str` and convert to UTC timestamp.

    Arguments:
        time_str: ISO8601 format timestamp, example: 2011-07-06T21:54:23.000-07:00.

    Raises:
        binding.HTTPError: rest client returns an exception (everything
            else than 400 code).
        InvalidTimeFormatException: when time format is invalid (rest
            client returns 400 code).

    Returns:
        UTC timestamp.
    """

    return datetime.datetime.utcfromtimestamp(self.to_seconds(time_str))

timer_queue.py

A simple thread safe timer queue implementation which has O(logn) time complexity.

TEARDOWN_SENTINEL = None module-attribute

__all__ = ['Timer', 'TimerQueueStruct', 'TimerQueue'] module-attribute

Timer

Timer wraps the callback and timestamp related attributes.

Source code in solnlib/timer_queue.py 
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
class Timer:
    """Timer wraps the callback and timestamp related attributes."""

    _ident = 0
    _lock = threading.Lock()

    def __init__(self, callback: Callable, when: int, interval: int, ident: int = None):
        """Initializes Timer.

        Arguments:
            callback: Arbitrary callable object.
            when: The first expiration time, seconds since epoch.
            interval: Timer interval, if equals 0, one time timer, otherwise
                the timer will be periodically executed.
            ident: (optional) Timer identity.
        """
        self._callback = callback
        self.when = when
        self.interval = interval

        if ident is not None:
            self.ident = ident
        else:
            with Timer._lock:
                self.ident = Timer._ident + 1
                Timer._ident = Timer._ident + 1

    def update_expiration(self):
        self.when += self.interval

    def __hash__(self):
        return hash(self.ident)

    def __eq__(self, other):
        return isinstance(other, Timer) and (self.ident == other.ident)

    def __lt__(self, other):
        return (self.when, self.ident) < (other.when, other.ident)

    def __le__(self, other):
        return (self.when, self.ident) <= (other.when, other.ident)

    def __gt__(self, other):
        return (self.when, self.ident) > (other.when, other.ident)

    def __ge__(self, other):
        return (self.when, self.ident) >= (other.when, other.ident)

    def __call__(self):
        self._callback()

ident = Timer._ident + 1 instance-attribute

interval = interval instance-attribute

when = when instance-attribute

__call__()

Source code in solnlib/timer_queue.py 
79
80
def __call__(self):
    self._callback()

__eq__(other)

Source code in solnlib/timer_queue.py 
64
65
def __eq__(self, other):
    return isinstance(other, Timer) and (self.ident == other.ident)

__ge__(other)

Source code in solnlib/timer_queue.py 
76
77
def __ge__(self, other):
    return (self.when, self.ident) >= (other.when, other.ident)

__gt__(other)

Source code in solnlib/timer_queue.py 
73
74
def __gt__(self, other):
    return (self.when, self.ident) > (other.when, other.ident)

__hash__()

Source code in solnlib/timer_queue.py 
61
62
def __hash__(self):
    return hash(self.ident)

__init__(callback, when, interval, ident=None)

Initializes Timer.

Parameters:

Name Type Description Default
callback Callable

Arbitrary callable object.

 required
when int

The first expiration time, seconds since epoch.

 required
interval int

Timer interval, if equals 0, one time timer, otherwise the timer will be periodically executed.

 required
ident int

(optional) Timer identity.

 None
Source code in solnlib/timer_queue.py 
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
def __init__(self, callback: Callable, when: int, interval: int, ident: int = None):
    """Initializes Timer.

    Arguments:
        callback: Arbitrary callable object.
        when: The first expiration time, seconds since epoch.
        interval: Timer interval, if equals 0, one time timer, otherwise
            the timer will be periodically executed.
        ident: (optional) Timer identity.
    """
    self._callback = callback
    self.when = when
    self.interval = interval

    if ident is not None:
        self.ident = ident
    else:
        with Timer._lock:
            self.ident = Timer._ident + 1
            Timer._ident = Timer._ident + 1

__le__(other)

Source code in solnlib/timer_queue.py 
70
71
def __le__(self, other):
    return (self.when, self.ident) <= (other.when, other.ident)

__lt__(other)

Source code in solnlib/timer_queue.py 
67
68
def __lt__(self, other):
    return (self.when, self.ident) < (other.when, other.ident)

update_expiration()

Source code in solnlib/timer_queue.py 
58
59
def update_expiration(self):
    self.when += self.interval

TimerQueue

A simple timer queue implementation.

It runs a separate thread to handle timers Note: to effectively use this timer queue, the timer callback should be short, otherwise it will cause other timers’s delay execution. A typical use scenario in production is that the timers are just a simple functions which inject themselvies to a task queue and then they are picked up by a threading/process pool to execute, as shows below:

Timers --enqueue---> TimerQueue --------expiration-----------
                                                            |
                                                            |
                                                           \|/
Threading/Process Pool <---- TaskQueue <--enqueue-- Timers' callback (nonblocking)

Examples:

>>> from solnlib import timer_queue
>>> tq = timer_queue.TimerQueue()
>>> tq.start()
>>> t = tq.add_timer(my_func, time.time(), 10)
>>> # do other stuff
>>> tq.stop()
Source code in solnlib/timer_queue.py 
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
class TimerQueue:
    r"""A simple timer queue implementation.

    It runs a separate thread to handle timers Note: to effectively use this
    timer queue, the timer callback should be short, otherwise it will cause
    other timers's delay execution. A typical use scenario in production is
    that the timers are just a simple functions which inject themselvies to
    a task queue and then they are picked up by a threading/process pool to
    execute, as shows below:

        Timers --enqueue---> TimerQueue --------expiration-----------
                                                                    |
                                                                    |
                                                                   \|/
        Threading/Process Pool <---- TaskQueue <--enqueue-- Timers' callback (nonblocking)

    Examples:
           >>> from solnlib import timer_queue
           >>> tq = timer_queue.TimerQueue()
           >>> tq.start()
           >>> t = tq.add_timer(my_func, time.time(), 10)
           >>> # do other stuff
           >>> tq.stop()
    """

    def __init__(self):
        self._timers = TimerQueueStruct()
        self._lock = threading.Lock()
        self._wakeup_queue = Queue.Queue()
        self._thr = threading.Thread(target=self._check_and_execute)
        self._thr.daemon = True
        self._started = False

    def start(self):
        """Start the timer queue."""

        if self._started:
            return
        self._started = True

        self._thr.start()
        logging.info("TimerQueue started.")

    def stop(self):
        """Stop the timer queue."""

        if not self._started:
            return
        self._started = True

        self._wakeup(TEARDOWN_SENTINEL)
        self._thr.join()

    def add_timer(
        self, callback: Callable, when: int, interval: int, ident: int = None
    ) -> Timer:
        """Add timer to the queue.

        Arguments:
            callback: Arbitrary callable object.
            when: The first expiration time, seconds since epoch.
            interval: Timer interval, if equals 0, one time timer, otherwise
                the timer will be periodically executed
            ident: (optional) Timer identity.

        Returns:
            A timer object which should not be manipulated directly by
                clients. Used to delete/update the timer.
        """

        with self._lock:
            timer = self._timers.add_timer(callback, when, interval, ident)
        self._wakeup()
        return timer

    def remove_timer(self, timer: Timer):
        """Remove timer from the queue.

        Arguments:
            timer: Timer object to remove.
        """

        with self._lock:
            self._timers.remove_timer(timer)

    def _check_and_execute(self):
        wakeup_queue = self._wakeup_queue
        while 1:
            (next_expired_time, expired_timers) = self._get_expired_timers()
            for timer in expired_timers:
                try:
                    # Note, please make timer callback effective/short
                    timer()
                except Exception:
                    logging.error(traceback.format_exc())

            self._reset_timers(expired_timers)

            sleep_time = _calc_sleep_time(next_expired_time)
            try:
                wakeup = wakeup_queue.get(timeout=sleep_time)
                if wakeup is TEARDOWN_SENTINEL:
                    break
            except Queue.Empty:
                pass
        logging.info("TimerQueue stopped.")

    def _get_expired_timers(self):
        with self._lock:
            return self._timers.get_expired_timers()

    def _reset_timers(self, expired_timers):
        with self._lock:
            has_new_timer = self._timers.reset_timers(expired_timers)

        if has_new_timer:
            self._wakeup()

    def _wakeup(self, something="not_None"):
        self._wakeup_queue.put(something)

__init__()

Source code in solnlib/timer_queue.py 
218
219
220
221
222
223
224
def __init__(self):
    self._timers = TimerQueueStruct()
    self._lock = threading.Lock()
    self._wakeup_queue = Queue.Queue()
    self._thr = threading.Thread(target=self._check_and_execute)
    self._thr.daemon = True
    self._started = False

add_timer(callback, when, interval, ident=None)

Add timer to the queue.

Parameters:

Name Type Description Default
callback Callable

Arbitrary callable object.

 required
when int

The first expiration time, seconds since epoch.

 required
interval int

Timer interval, if equals 0, one time timer, otherwise the timer will be periodically executed

 required
ident int

(optional) Timer identity.

 None

Returns:

Type Description
Timer

A timer object which should not be manipulated directly by clients. Used to delete/update the timer.
Source code in solnlib/timer_queue.py 
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
def add_timer(
    self, callback: Callable, when: int, interval: int, ident: int = None
) -> Timer:
    """Add timer to the queue.

    Arguments:
        callback: Arbitrary callable object.
        when: The first expiration time, seconds since epoch.
        interval: Timer interval, if equals 0, one time timer, otherwise
            the timer will be periodically executed
        ident: (optional) Timer identity.

    Returns:
        A timer object which should not be manipulated directly by
            clients. Used to delete/update the timer.
    """

    with self._lock:
        timer = self._timers.add_timer(callback, when, interval, ident)
    self._wakeup()
    return timer

remove_timer(timer)

Remove timer from the queue.

Parameters:

Name Type Description Default
timer Timer

Timer object to remove.

 required
Source code in solnlib/timer_queue.py 
268
269
270
271
272
273
274
275
276
def remove_timer(self, timer: Timer):
    """Remove timer from the queue.

    Arguments:
        timer: Timer object to remove.
    """

    with self._lock:
        self._timers.remove_timer(timer)

start()

Start the timer queue.

Source code in solnlib/timer_queue.py 
226
227
228
229
230
231
232
233
234
def start(self):
    """Start the timer queue."""

    if self._started:
        return
    self._started = True

    self._thr.start()
    logging.info("TimerQueue started.")

stop()

Stop the timer queue.

Source code in solnlib/timer_queue.py 
236
237
238
239
240
241
242
243
244
def stop(self):
    """Stop the timer queue."""

    if not self._started:
        return
    self._started = True

    self._wakeup(TEARDOWN_SENTINEL)
    self._thr.join()

TimerQueueStruct

The underlying data structure for TimerQueue.

Source code in solnlib/timer_queue.py 
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
class TimerQueueStruct:
    """The underlying data structure for TimerQueue."""

    def __init__(self):
        self._timers = sc.SortedSet()
        self._cancelling_timers = {}

    def add_timer(
        self, callback: Callable, when: int, interval: int, ident: int
    ) -> Timer:
        """Add timer to the data structure.

        Arguments:
            callback: Arbitrary callable object.
            when: The first expiration time, seconds since epoch.
            interval: Timer interval, if equals 0, one time timer, otherwise
                the timer will be periodically executed
            ident: (optional) Timer identity.

        Returns:
            A timer object which should not be manipulated directly by
                clients. Used to delete/update the timer.
        """

        timer = Timer(callback, when, interval, ident)
        self._timers.add(timer)
        return timer

    def remove_timer(self, timer: Timer):
        """Remove timer from data structure.

        Arguments:
            timer: Timer object which is returned by ``TimerQueueStruct.add_timer``.
        """

        try:
            self._timers.remove(timer)
        except ValueError:
            logging.info(
                "Timer=%s is not in queue, move it to cancelling " "list", timer.ident
            )
        else:
            self._cancelling_timers[timer.ident] = timer

    def get_expired_timers(self) -> Tuple:
        """Get a list of expired timers.

        Returns:
            A tuple of ``Timer``, empty list if there is no expired timers.
        """

        next_expired_time = 0
        now = time()
        expired_timers = []
        for timer in self._timers:
            if timer.when <= now:
                expired_timers.append(timer)

        if expired_timers:
            del self._timers[: len(expired_timers)]

        if self._timers:
            next_expired_time = self._timers[0].when
        return next_expired_time, expired_timers

    def reset_timers(self, expired_timers: List[Timer]) -> bool:
        """Re-add the expired periodical timers to data structure for next
        round scheduling.

        Arguments:
            expired_timers: List of expired timers.

        Returns:
            True if there are timers added, False otherwise.
        """

        has_new_timer = False
        cancelling_timers = self._cancelling_timers
        for timer in expired_timers:
            if timer.ident in cancelling_timers:
                continue
            elif timer.interval:
                # Repeated timer
                timer.update_expiration()
                self._timers.add(timer)
                has_new_timer = True
        cancelling_timers.clear()
        return has_new_timer

    def check_and_execute(self) -> float:
        """Get expired timers and execute callbacks for the timers.

        Returns:
            Duration of next expired timer.
        """

        (next_expired_time, expired_timers) = self.get_expired_timers()
        for timer in expired_timers:
            try:
                timer()
            except Exception:
                logging.error(traceback.format_exc())

        self.reset_timers(expired_timers)
        return _calc_sleep_time(next_expired_time)

__init__()

Source code in solnlib/timer_queue.py 
89
90
91
def __init__(self):
    self._timers = sc.SortedSet()
    self._cancelling_timers = {}

add_timer(callback, when, interval, ident)

Add timer to the data structure.

Parameters:

Name Type Description Default
callback Callable

Arbitrary callable object.

 required
when int

The first expiration time, seconds since epoch.

 required
interval int

Timer interval, if equals 0, one time timer, otherwise the timer will be periodically executed

 required
ident int

(optional) Timer identity.

 required

Returns:

Type Description
Timer

A timer object which should not be manipulated directly by clients. Used to delete/update the timer.
Source code in solnlib/timer_queue.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
def add_timer(
    self, callback: Callable, when: int, interval: int, ident: int
) -> Timer:
    """Add timer to the data structure.

    Arguments:
        callback: Arbitrary callable object.
        when: The first expiration time, seconds since epoch.
        interval: Timer interval, if equals 0, one time timer, otherwise
            the timer will be periodically executed
        ident: (optional) Timer identity.

    Returns:
        A timer object which should not be manipulated directly by
            clients. Used to delete/update the timer.
    """

    timer = Timer(callback, when, interval, ident)
    self._timers.add(timer)
    return timer

check_and_execute()

Get expired timers and execute callbacks for the timers.

Returns:

Type Description
float

Duration of next expired timer.
Source code in solnlib/timer_queue.py 
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
def check_and_execute(self) -> float:
    """Get expired timers and execute callbacks for the timers.

    Returns:
        Duration of next expired timer.
    """

    (next_expired_time, expired_timers) = self.get_expired_timers()
    for timer in expired_timers:
        try:
            timer()
        except Exception:
            logging.error(traceback.format_exc())

    self.reset_timers(expired_timers)
    return _calc_sleep_time(next_expired_time)

get_expired_timers()

Get a list of expired timers.

Returns:

Type Description
Tuple

A tuple of Timer, empty list if there is no expired timers.
Source code in solnlib/timer_queue.py 
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
def get_expired_timers(self) -> Tuple:
    """Get a list of expired timers.

    Returns:
        A tuple of ``Timer``, empty list if there is no expired timers.
    """

    next_expired_time = 0
    now = time()
    expired_timers = []
    for timer in self._timers:
        if timer.when <= now:
            expired_timers.append(timer)

    if expired_timers:
        del self._timers[: len(expired_timers)]

    if self._timers:
        next_expired_time = self._timers[0].when
    return next_expired_time, expired_timers

remove_timer(timer)

Remove timer from data structure.

Parameters:

Name Type Description Default
timer Timer

Timer object which is returned by TimerQueueStruct.add_timer.

 required
Source code in solnlib/timer_queue.py 
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
def remove_timer(self, timer: Timer):
    """Remove timer from data structure.

    Arguments:
        timer: Timer object which is returned by ``TimerQueueStruct.add_timer``.
    """

    try:
        self._timers.remove(timer)
    except ValueError:
        logging.info(
            "Timer=%s is not in queue, move it to cancelling " "list", timer.ident
        )
    else:
        self._cancelling_timers[timer.ident] = timer

reset_timers(expired_timers)

Re-add the expired periodical timers to data structure for next round scheduling.

Parameters:

Name Type Description Default
expired_timers List[Timer]

List of expired timers.

 required

Returns:

Type Description
bool

True if there are timers added, False otherwise.
Source code in solnlib/timer_queue.py 
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
def reset_timers(self, expired_timers: List[Timer]) -> bool:
    """Re-add the expired periodical timers to data structure for next
    round scheduling.

    Arguments:
        expired_timers: List of expired timers.

    Returns:
        True if there are timers added, False otherwise.
    """

    has_new_timer = False
    cancelling_timers = self._cancelling_timers
    for timer in expired_timers:
        if timer.ident in cancelling_timers:
            continue
        elif timer.interval:
            # Repeated timer
            timer.update_expiration()
            self._timers.add(timer)
            has_new_timer = True
    cancelling_timers.clear()
    return has_new_timer

user_access.py

Splunk user access control related utilities.

__all__ = ['ObjectACLException', 'ObjectACL', 'ObjectACLManagerException', 'ObjectACLManager', 'AppCapabilityManagerException', 'AppCapabilityManager', 'UserAccessException', 'check_user_access', 'InvalidSessionKeyException', 'get_current_username', 'UserNotExistException', 'get_user_capabilities', 'user_is_capable', 'get_user_roles'] module-attribute

AppCapabilityManager

App capability manager.

Examples:

>>> from solnlib import user_access
>>> acm = user_access.AppCapabilityManager('test_collection',
                                           session_key,
                                           'Splunk_TA_test')
>>> acm.register_capabilities(...)
>>> acm.unregister_capabilities(...)
Source code in solnlib/user_access.py 
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
class AppCapabilityManager:
    """App capability manager.

    Examples:
       >>> from solnlib import user_access
       >>> acm = user_access.AppCapabilityManager('test_collection',
                                                  session_key,
                                                  'Splunk_TA_test')
       >>> acm.register_capabilities(...)
       >>> acm.unregister_capabilities(...)
    """

    def __init__(
        self,
        collection_name: str,
        session_key: str,
        app: str,
        owner: str = "nobody",
        scheme: str = None,
        host: str = None,
        port: int = None,
        **context: dict,
    ):
        """Initializes AppCapabilityManager.

        Arguments:
            collection_name: Collection name to store capabilities.
            session_key: Splunk access token.
            app: App name of namespace.
            owner: (optional) Owner of namespace, default is `nobody`.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            context: Other configurations for Splunk rest client.

        Raises:
            AppCapabilityManagerException: If init AppCapabilityManager failed.
        """
        self._app = app

        collection_name = f"{app}_{collection_name}"
        try:
            self._collection_data = _utils.get_collection_data(
                collection_name,
                session_key,
                app,
                owner,
                scheme,
                host,
                port,
                None,
                **context,
            )
        except KeyError:
            raise AppCapabilityManagerException(
                f"Get app capabilities collection: {collection_name} failed."
            )

    @utils.retry(exceptions=[binding.HTTPError])
    def register_capabilities(self, capabilities: dict):
        """Register app capabilities.

        Arguments:
            capabilities: App capabilities, example:

                {
                    'object_type1': {
                        'read': 'read_app_object_type1',
                        'write': 'write_app_object_type1',
                        'delete': 'delete_app_object_type1'},
                        'object_type2': {
                        'read': 'read_app_object_type2',
                        'write': 'write_app_object_type2',
                        'delete': 'delete_app_object_type2'
                    },
                    ...
                }
        """

        record = {"_key": self._app, "capabilities": capabilities}
        self._collection_data.batch_save(record)

    @utils.retry(exceptions=[binding.HTTPError])
    def unregister_capabilities(self):
        """Unregister app capabilities.

        Raises:
            AppCapabilityNotExistException: If app capabilities are not registered.
        """

        try:
            self._collection_data.delete_by_id(self._app)
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            raise AppCapabilityNotExistException(
                "App capabilities for %s have not been registered." % self._app
            )

    @utils.retry(exceptions=[binding.HTTPError])
    def capabilities_are_registered(self) -> bool:
        """Check if app capabilities are registered.

        Returns:
            True if app capabilities are registered else False.
        """

        try:
            self._collection_data.query_by_id(self._app)
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            return False

        return True

    @utils.retry(exceptions=[binding.HTTPError])
    def get_capabilities(self) -> dict:
        """Get app capabilities.

        Returns:
            App capabilities.

        Raises:
             AppCapabilityNotExistException: If app capabilities are not registered.
        """

        try:
            record = self._collection_data.query_by_id(self._app)
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            raise AppCapabilityNotExistException(
                "App capabilities for %s have not been registered." % self._app
            )

        return record["capabilities"]

__init__(collection_name, session_key, app, owner='nobody', scheme=None, host=None, port=None, **context)

Initializes AppCapabilityManager.

Parameters:

Name Type Description Default
collection_name str

Collection name to store capabilities.

 required
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner str

(optional) Owner of namespace, default is nobody.

 'nobody'
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}

Raises:

Type Description
AppCapabilityManagerException

If init AppCapabilityManager failed.
Source code in solnlib/user_access.py 
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
def __init__(
    self,
    collection_name: str,
    session_key: str,
    app: str,
    owner: str = "nobody",
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
):
    """Initializes AppCapabilityManager.

    Arguments:
        collection_name: Collection name to store capabilities.
        session_key: Splunk access token.
        app: App name of namespace.
        owner: (optional) Owner of namespace, default is `nobody`.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Raises:
        AppCapabilityManagerException: If init AppCapabilityManager failed.
    """
    self._app = app

    collection_name = f"{app}_{collection_name}"
    try:
        self._collection_data = _utils.get_collection_data(
            collection_name,
            session_key,
            app,
            owner,
            scheme,
            host,
            port,
            None,
            **context,
        )
    except KeyError:
        raise AppCapabilityManagerException(
            f"Get app capabilities collection: {collection_name} failed."
        )

capabilities_are_registered()

Check if app capabilities are registered.

Returns:

Type Description
bool

True if app capabilities are registered else False.
Source code in solnlib/user_access.py 
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
@utils.retry(exceptions=[binding.HTTPError])
def capabilities_are_registered(self) -> bool:
    """Check if app capabilities are registered.

    Returns:
        True if app capabilities are registered else False.
    """

    try:
        self._collection_data.query_by_id(self._app)
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        return False

    return True

get_capabilities()

Get app capabilities.

Returns:

Type Description
dict

App capabilities.

Raises:

Type Description
AppCapabilityNotExistException

If app capabilities are not registered.
Source code in solnlib/user_access.py 
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
@utils.retry(exceptions=[binding.HTTPError])
def get_capabilities(self) -> dict:
    """Get app capabilities.

    Returns:
        App capabilities.

    Raises:
         AppCapabilityNotExistException: If app capabilities are not registered.
    """

    try:
        record = self._collection_data.query_by_id(self._app)
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise AppCapabilityNotExistException(
            "App capabilities for %s have not been registered." % self._app
        )

    return record["capabilities"]

register_capabilities(capabilities)

Register app capabilities.

Parameters:

Name Type Description Default
capabilities dict

App capabilities, example:

{ ‘object_type1’: { ‘read’: ‘read_app_object_type1’, ‘write’: ‘write_app_object_type1’, ‘delete’: ‘delete_app_object_type1’}, ‘object_type2’: { ‘read’: ‘read_app_object_type2’, ‘write’: ‘write_app_object_type2’, ‘delete’: ‘delete_app_object_type2’ }, … }

 required
Source code in solnlib/user_access.py 
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
@utils.retry(exceptions=[binding.HTTPError])
def register_capabilities(self, capabilities: dict):
    """Register app capabilities.

    Arguments:
        capabilities: App capabilities, example:

            {
                'object_type1': {
                    'read': 'read_app_object_type1',
                    'write': 'write_app_object_type1',
                    'delete': 'delete_app_object_type1'},
                    'object_type2': {
                    'read': 'read_app_object_type2',
                    'write': 'write_app_object_type2',
                    'delete': 'delete_app_object_type2'
                },
                ...
            }
    """

    record = {"_key": self._app, "capabilities": capabilities}
    self._collection_data.batch_save(record)

unregister_capabilities()

Unregister app capabilities.

Raises:

Type Description
AppCapabilityNotExistException

If app capabilities are not registered.
Source code in solnlib/user_access.py 
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
@utils.retry(exceptions=[binding.HTTPError])
def unregister_capabilities(self):
    """Unregister app capabilities.

    Raises:
        AppCapabilityNotExistException: If app capabilities are not registered.
    """

    try:
        self._collection_data.delete_by_id(self._app)
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise AppCapabilityNotExistException(
            "App capabilities for %s have not been registered." % self._app
        )

AppCapabilityManagerException

Bases: Exception

Exception for AppCapabilityManager.

Source code in solnlib/user_access.py 
542
543
544
545
class AppCapabilityManagerException(Exception):
    """Exception for AppCapabilityManager."""

    pass

AppCapabilityNotExistException

Bases: Exception

Exception for the situation when AppCapability does not exist for a specific app.

Source code in solnlib/user_access.py 
548
549
550
551
552
class AppCapabilityNotExistException(Exception):
    """Exception for the situation when AppCapability does not exist for a
    specific app."""

    pass

InvalidSessionKeyException

Bases: Exception

Exception when Splunk session key is invalid.

Source code in solnlib/user_access.py 
773
774
775
776
class InvalidSessionKeyException(Exception):
    """Exception when Splunk session key is invalid."""

    pass

ObjectACL

Object ACL record.

Examples:

>>> from solnlib import user_access
>>> obj_acl = user_access.ObjectACL(
>>>    'test_collection',
>>>    '9defa6f510d711e6be16a45e60e34295',
>>>    'test_object',
>>>    'Splunk_TA_test',
>>>    'admin',
>>>    {'read': ['*'], 'write': ['admin'], 'delete': ['admin']},
>>>    False)
Source code in solnlib/user_access.py 
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
class ObjectACL:
    """Object ACL record.

    Examples:
       >>> from solnlib import user_access
       >>> obj_acl = user_access.ObjectACL(
       >>>    'test_collection',
       >>>    '9defa6f510d711e6be16a45e60e34295',
       >>>    'test_object',
       >>>    'Splunk_TA_test',
       >>>    'admin',
       >>>    {'read': ['*'], 'write': ['admin'], 'delete': ['admin']},
       >>>    False)
    """

    OBJ_COLLECTION_KEY = "obj_collection"
    OBJ_ID_KEY = "obj_id"
    OBJ_TYPE_KEY = "obj_type"
    OBJ_APP_KEY = "obj_app"
    OBJ_OWNER_KEY = "obj_owner"
    OBJ_PERMS_KEY = "obj_perms"
    OBJ_PERMS_READ_KEY = "read"
    OBJ_PERMS_WRITE_KEY = "write"
    OBJ_PERMS_DELETE_KEY = "delete"
    OBJ_PERMS_ALLOW_ALL = "*"
    OBJ_SHARED_BY_INCLUSION_KEY = "obj_shared_by_inclusion"

    def __init__(
        self,
        obj_collection: str,
        obj_id: str,
        obj_type: str,
        obj_app: str,
        obj_owner: str,
        obj_perms: dict,
        obj_shared_by_inclusion: bool,
    ):
        """Initializes ObjectACL.

        Arguments:
            obj_collection: Collection where object currently stored.
            obj_id: ID of this object.
            obj_type: Type of this object.
            obj_app: App of this object.
            obj_owner: Owner of this object.
            obj_perms: Object perms, like: {'read': ['*'], 'write': ['admin'], 'delete': ['admin']}.
            obj_shared_by_inclusion: Flag of object is shared by inclusion.
        """
        self.obj_collection = obj_collection
        self.obj_id = obj_id
        self.obj_type = obj_type
        self.obj_app = obj_app
        self.obj_owner = obj_owner
        self._check_perms(obj_perms)
        self._obj_perms = obj_perms
        self.obj_shared_by_inclusion = obj_shared_by_inclusion

    @classmethod
    def _check_perms(cls, obj_perms):
        if not isinstance(obj_perms, dict):
            raise ObjectACLException(
                "Invalid object acl perms type: %s, should be a dict." % type(obj_perms)
            )

        if not (
            cls.OBJ_PERMS_READ_KEY in obj_perms
            and cls.OBJ_PERMS_WRITE_KEY in obj_perms
            and cls.OBJ_PERMS_DELETE_KEY in obj_perms
        ):
            raise ObjectACLException(
                "Invalid object acl perms: %s, "
                "should include read, write and delete perms." % obj_perms
            )

    @property
    def obj_perms(self):
        return self._obj_perms

    @obj_perms.setter
    def obj_perms(self, obj_perms):
        self._check_perms(obj_perms)
        self._obj_perms = obj_perms

    @property
    def record(self) -> dict:
        """Get object acl record.

        Returns: Object acl record, like:

            {
                '_key': 'test_collection-1234',
                'obj_collection': 'test_collection',
                'obj_id': '1234',
                'obj_type': 'test_object',
                'obj_app': 'Splunk_TA_test',
                'obj_owner': 'admin',
                'obj_perms': {'read': ['*'], 'write': ['admin'], 'delete': ['admin']},
                'obj_shared_by_inclusion': True
            }
        """

        return {
            "_key": self.generate_key(self.obj_collection, self.obj_id),
            self.OBJ_COLLECTION_KEY: self.obj_collection,
            self.OBJ_ID_KEY: self.obj_id,
            self.OBJ_TYPE_KEY: self.obj_type,
            self.OBJ_APP_KEY: self.obj_app,
            self.OBJ_OWNER_KEY: self.obj_owner,
            self.OBJ_PERMS_KEY: self._obj_perms,
            self.OBJ_SHARED_BY_INCLUSION_KEY: self.obj_shared_by_inclusion,
        }

    @staticmethod
    def generate_key(obj_collection: str, obj_id: str) -> str:
        """Generate object acl record key.

        Arguments:
            obj_collection: Collection where object currently stored.
            obj_id: ID of this object.

        Returns:
            Object acl record key.
        """

        return "{obj_collection}_{obj_id}".format(
            obj_collection=obj_collection, obj_id=obj_id
        )

    @staticmethod
    def parse(obj_acl_record: dict) -> "ObjectACL":
        """Parse object acl record and construct a new `ObjectACL` object from
        it.

        Arguments:
            obj_acl_record: Object acl record.

        Returns:
            New `ObjectACL` object.
        """

        return ObjectACL(
            obj_acl_record[ObjectACL.OBJ_COLLECTION_KEY],
            obj_acl_record[ObjectACL.OBJ_ID_KEY],
            obj_acl_record[ObjectACL.OBJ_TYPE_KEY],
            obj_acl_record[ObjectACL.OBJ_APP_KEY],
            obj_acl_record[ObjectACL.OBJ_OWNER_KEY],
            obj_acl_record[ObjectACL.OBJ_PERMS_KEY],
            obj_acl_record[ObjectACL.OBJ_SHARED_BY_INCLUSION_KEY],
        )

    def merge(self, obj_acl: "ObjectACL"):
        """Merge current object perms with perms of `obj_acl`.

        Arguments:
            obj_acl: Object acl to merge.
        """

        for perm_key in self._obj_perms:
            self._obj_perms[perm_key] = list(
                set.union(
                    set(self._obj_perms[perm_key]), set(obj_acl._obj_perms[perm_key])
                )
            )
            if self.OBJ_PERMS_ALLOW_ALL in self._obj_perms[perm_key]:
                self._obj_perms[perm_key] = [self.OBJ_PERMS_ALLOW_ALL]

    def __str__(self):
        return json.dumps(self.record)

OBJ_APP_KEY = 'obj_app' instance-attribute class-attribute

OBJ_COLLECTION_KEY = 'obj_collection' instance-attribute class-attribute

OBJ_ID_KEY = 'obj_id' instance-attribute class-attribute

OBJ_OWNER_KEY = 'obj_owner' instance-attribute class-attribute

OBJ_PERMS_ALLOW_ALL = '*' instance-attribute class-attribute

OBJ_PERMS_DELETE_KEY = 'delete' instance-attribute class-attribute

OBJ_PERMS_KEY = 'obj_perms' instance-attribute class-attribute

OBJ_PERMS_READ_KEY = 'read' instance-attribute class-attribute

OBJ_PERMS_WRITE_KEY = 'write' instance-attribute class-attribute

OBJ_SHARED_BY_INCLUSION_KEY = 'obj_shared_by_inclusion' instance-attribute class-attribute

OBJ_TYPE_KEY = 'obj_type' instance-attribute class-attribute

obj_app = obj_app instance-attribute

obj_collection = obj_collection instance-attribute

obj_id = obj_id instance-attribute

obj_owner = obj_owner instance-attribute

obj_perms property writable

obj_shared_by_inclusion = obj_shared_by_inclusion instance-attribute

obj_type = obj_type instance-attribute

record: dict property

Get object acl record.

Object acl record, like:

Type Description
dict

{ ‘_key’: ‘test_collection-1234’, ‘obj_collection’: ‘test_collection’, ‘obj_id’: ‘1234’, ‘obj_type’: ‘test_object’, ‘obj_app’: ‘Splunk_TA_test’, ‘obj_owner’: ‘admin’, ‘obj_perms’: {‘read’: [‘*’], ‘write’: [‘admin’], ‘delete’: [‘admin’]}, ‘obj_shared_by_inclusion’: True
dict

}

__init__(obj_collection, obj_id, obj_type, obj_app, obj_owner, obj_perms, obj_shared_by_inclusion)

Initializes ObjectACL.

Parameters:

Name Type Description Default
obj_collection str

Collection where object currently stored.

 required
obj_id str

ID of this object.

 required
obj_type str

Type of this object.

 required
obj_app str

App of this object.

 required
obj_owner str

Owner of this object.

 required
obj_perms dict

Object perms, like: {‘read’: [‘*’], ‘write’: [‘admin’], ‘delete’: [‘admin’]}.

 required
obj_shared_by_inclusion bool

Flag of object is shared by inclusion.

 required
Source code in solnlib/user_access.py 
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
def __init__(
    self,
    obj_collection: str,
    obj_id: str,
    obj_type: str,
    obj_app: str,
    obj_owner: str,
    obj_perms: dict,
    obj_shared_by_inclusion: bool,
):
    """Initializes ObjectACL.

    Arguments:
        obj_collection: Collection where object currently stored.
        obj_id: ID of this object.
        obj_type: Type of this object.
        obj_app: App of this object.
        obj_owner: Owner of this object.
        obj_perms: Object perms, like: {'read': ['*'], 'write': ['admin'], 'delete': ['admin']}.
        obj_shared_by_inclusion: Flag of object is shared by inclusion.
    """
    self.obj_collection = obj_collection
    self.obj_id = obj_id
    self.obj_type = obj_type
    self.obj_app = obj_app
    self.obj_owner = obj_owner
    self._check_perms(obj_perms)
    self._obj_perms = obj_perms
    self.obj_shared_by_inclusion = obj_shared_by_inclusion

__str__()

Source code in solnlib/user_access.py 
216
217
def __str__(self):
    return json.dumps(self.record)

generate_key(obj_collection, obj_id) staticmethod

Generate object acl record key.

Parameters:

Name Type Description Default
obj_collection str

Collection where object currently stored.

 required
obj_id str

ID of this object.

 required

Returns:

Type Description
str

Object acl record key.
Source code in solnlib/user_access.py 
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
@staticmethod
def generate_key(obj_collection: str, obj_id: str) -> str:
    """Generate object acl record key.

    Arguments:
        obj_collection: Collection where object currently stored.
        obj_id: ID of this object.

    Returns:
        Object acl record key.
    """

    return "{obj_collection}_{obj_id}".format(
        obj_collection=obj_collection, obj_id=obj_id
    )

merge(obj_acl)

Merge current object perms with perms of obj_acl.

Parameters:

Name Type Description Default
obj_acl ObjectACL

Object acl to merge.

 required
Source code in solnlib/user_access.py 
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
def merge(self, obj_acl: "ObjectACL"):
    """Merge current object perms with perms of `obj_acl`.

    Arguments:
        obj_acl: Object acl to merge.
    """

    for perm_key in self._obj_perms:
        self._obj_perms[perm_key] = list(
            set.union(
                set(self._obj_perms[perm_key]), set(obj_acl._obj_perms[perm_key])
            )
        )
        if self.OBJ_PERMS_ALLOW_ALL in self._obj_perms[perm_key]:
            self._obj_perms[perm_key] = [self.OBJ_PERMS_ALLOW_ALL]

parse(obj_acl_record) staticmethod

Parse object acl record and construct a new ObjectACL object from it.

Parameters:

Name Type Description Default
obj_acl_record dict

Object acl record.

 required

Returns:

Type Description
ObjectACL

New ObjectACL object.
Source code in solnlib/user_access.py 
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
@staticmethod
def parse(obj_acl_record: dict) -> "ObjectACL":
    """Parse object acl record and construct a new `ObjectACL` object from
    it.

    Arguments:
        obj_acl_record: Object acl record.

    Returns:
        New `ObjectACL` object.
    """

    return ObjectACL(
        obj_acl_record[ObjectACL.OBJ_COLLECTION_KEY],
        obj_acl_record[ObjectACL.OBJ_ID_KEY],
        obj_acl_record[ObjectACL.OBJ_TYPE_KEY],
        obj_acl_record[ObjectACL.OBJ_APP_KEY],
        obj_acl_record[ObjectACL.OBJ_OWNER_KEY],
        obj_acl_record[ObjectACL.OBJ_PERMS_KEY],
        obj_acl_record[ObjectACL.OBJ_SHARED_BY_INCLUSION_KEY],
    )

ObjectACLException

Bases: Exception

Source code in solnlib/user_access.py 
46
47
class ObjectACLException(Exception):
    pass

ObjectACLManager

Object ACL manager.

Examples:

>>> from solnlib import user_access
>>> oaclm = user_access.ObjectACLManager(session_key,
                                         'Splunk_TA_test')
Source code in solnlib/user_access.py 
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
class ObjectACLManager:
    """Object ACL manager.

    Examples:
       >>> from solnlib import user_access
       >>> oaclm = user_access.ObjectACLManager(session_key,
                                                'Splunk_TA_test')
    """

    def __init__(
        self,
        collection_name: str,
        session_key: str,
        app: str,
        owner: Optional[str] = "nobody",
        scheme: Optional[str] = None,
        host: Optional[str] = None,
        port: Optional[int] = None,
        **context: dict,
    ):
        """Initializes ObjectACLManager.

        Arguments:
            collection_name: Collection name to store object ACL info.
            session_key: Splunk access token.
            app: App name of namespace.
            owner: (optional) Owner of namespace, default is `nobody`.
            scheme: (optional) The access scheme, default is None.
            host: (optional) The host name, default is None.
            port: (optional) The port number, default is None.
            context: Other configurations for Splunk rest client.

        Raises:
            ObjectACLManagerException: If init ObjectACLManager failed.
        """
        collection_name = "{app}_{collection_name}".format(
            app=app, collection_name=collection_name
        )
        try:
            self._collection_data = _utils.get_collection_data(
                collection_name,
                session_key,
                app,
                owner,
                scheme,
                host,
                port,
                None,
                **context,
            )
        except KeyError:
            raise ObjectACLManagerException(
                f"Get object acl collection: {collection_name} fail."
            )

    @utils.retry(exceptions=[binding.HTTPError])
    def update_acl(
        self,
        obj_collection: str,
        obj_id: str,
        obj_type: str,
        obj_app: str,
        obj_owner: str,
        obj_perms: dict,
        obj_shared_by_inclusion: bool = True,
        replace_existing: bool = True,
    ):
        """Update acl info of object.

        Construct a new object acl info first, if `replace_existing` is True
        then replace existing acl info else merge new object acl info with the
        old one and replace the old acl info with merged acl info.

        Arguments:
            obj_collection: Collection where object currently stored.
            obj_id: ID of this object.
            obj_type: Type of this object.
            obj_app: App of this object.
            obj_owner: Owner of this object.
            obj_perms: Object perms, like:

                {
                    'read': ['*'],
                    'write': ['admin'],
                    'delete': ['admin']
                }.
            obj_shared_by_inclusion: (optional) Flag of object is shared by
                inclusion, default is True.
            replace_existing: (optional) Replace existing acl info flag, True
                indicates replace old acl info with new one else merge with old
                acl info, default is True.
        """

        obj_acl = ObjectACL(
            obj_collection,
            obj_id,
            obj_type,
            obj_app,
            obj_owner,
            obj_perms,
            obj_shared_by_inclusion,
        )

        if not replace_existing:
            try:
                old_obj_acl = self.get_acl(obj_collection, obj_id)
            except ObjectACLNotExistException:
                old_obj_acl = None

            if old_obj_acl:
                obj_acl.merge(old_obj_acl)

        self._collection_data.batch_save(obj_acl.record)

    @utils.retry(exceptions=[binding.HTTPError])
    def update_acls(
        self,
        obj_collection: str,
        obj_ids: List[str],
        obj_type: str,
        obj_app: str,
        obj_owner: str,
        obj_perms: dict,
        obj_shared_by_inclusion: bool = True,
        replace_existing: bool = True,
    ):
        """Batch update object acl info to all provided `obj_ids`.

        Arguments:
            obj_collection: Collection where objects currently stored.
            obj_ids: IDs list of objects.
            obj_type: Type of this object.
            obj_app: App of this object.
            obj_owner: Owner of this object.
            obj_perms: Object perms, like:

                {
                    'read': ['*'],
                    'write': ['admin'],
                    'delete': ['admin']
                }.
            obj_shared_by_inclusion: (optional) Flag of object is shared by
                inclusion, default is True.
            replace_existing: (optional) Replace existing acl info flag, True
                indicates replace old acl info with new one else merge with old acl
                info, default is True.
        """

        obj_acl_records = []
        for obj_id in obj_ids:
            obj_acl = ObjectACL(
                obj_collection,
                obj_id,
                obj_type,
                obj_app,
                obj_owner,
                obj_perms,
                obj_shared_by_inclusion,
            )

            if not replace_existing:
                try:
                    old_obj_acl = self.get_acl(obj_collection, obj_id)
                except ObjectACLNotExistException:
                    old_obj_acl = None

                if old_obj_acl:
                    obj_acl.merge(old_obj_acl)

            obj_acl_records.append(obj_acl.record)

        self._collection_data.batch_save(*obj_acl_records)

    @utils.retry(exceptions=[binding.HTTPError])
    def get_acl(self, obj_collection: str, obj_id: str) -> "ObjectACL":
        """Get acl info.

        Query object acl info with parameter of the combination of
        `obj_collection` and `obj_id` from `self.collection_name` and
        return it.

        Arguments:
            obj_collection: Collection where object currently stored.
            obj_id: ID of this object.

        Returns:
            Object acl info if success else None.

        Raises:
            ObjectACLNotExistException: If object ACL info does not exist.
        """

        key = ObjectACL.generate_key(obj_collection, obj_id)
        try:
            obj_acl = self._collection_data.query_by_id(key)
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            raise ObjectACLNotExistException(
                "Object ACL info of {}_{} does not exist.".format(
                    obj_collection, obj_id
                )
            )

        return ObjectACL.parse(obj_acl)

    @utils.retry(exceptions=[binding.HTTPError])
    def get_acls(self, obj_collection: str, obj_ids: List[str]) -> List[ObjectACL]:
        """Batch get acl info.

        Query objects acl info with parameter of the combination of
        `obj_collection` and `obj_ids` from KVStore and return them.

        Arguments:
            obj_collection: Collection where object currently stored.
            obj_ids: IDs of objects.

        Returns:
            List of `ObjectACL` instances.
        """

        query = json.dumps(
            {
                "$or": [
                    {"_key": ObjectACL.generate_key(obj_collection, obj_id)}
                    for obj_id in obj_ids
                ]
            }
        )
        obj_acls = self._collection_data.query(query=query)

        return [ObjectACL.parse(obj_acl) for obj_acl in obj_acls]

    @utils.retry(exceptions=[binding.HTTPError])
    def delete_acl(self, obj_collection: str, obj_id: str):
        """Delete acl info.

        Query object acl info with parameter of the combination of
        `obj_collection` and `obj_ids` from KVStore and delete it.

        Arguments:
            obj_collection: Collection where object currently stored.
            obj_id: ID of this object.

        Raises:
            ObjectACLNotExistException: If object ACL info does not exist.
        """

        key = ObjectACL.generate_key(obj_collection, obj_id)
        try:
            self._collection_data.delete_by_id(key)
        except binding.HTTPError as e:
            if e.status != 404:
                raise

            raise ObjectACLNotExistException(
                "Object ACL info of {}_{} does not exist.".format(
                    obj_collection, obj_id
                )
            )

    @utils.retry(exceptions=[binding.HTTPError])
    def delete_acls(self, obj_collection: str, obj_ids: List[str]):
        """Batch delete acl info.

        Query objects acl info with parameter of the combination of
        `obj_collection` and `obj_ids` from KVStore and delete them.

        Arguments:
            obj_collection: Collection where object currently stored.
            obj_ids: IDs of objects.
        """

        query = json.dumps(
            {
                "$or": [
                    {"_key": ObjectACL.generate_key(obj_collection, obj_id)}
                    for obj_id in obj_ids
                ]
            }
        )
        self._collection_data.delete(query=query)

    @utils.retry(exceptions=[binding.HTTPError])
    def get_accessible_object_ids(
        self, user: str, operation: str, obj_collection: str, obj_ids: List[str]
    ) -> List[str]:
        """Get accessible IDs of objects from `obj_acls`.

        Arguments:
            user: User name of current `operation`.
            operation: User operation, possible option: (read/write/delete).
            obj_collection: Collection where object currently stored.
            obj_ids: IDs of objects.

        Returns:
            List of IDs of accessible objects.
        """

        obj_acls = self.get_acls(obj_collection, obj_ids)
        accessible_obj_ids = []
        for obj_acl in obj_acls:
            perms = obj_acl.obj_perms[operation]
            if ObjectACL.OBJ_PERMS_ALLOW_ALL in perms or user in perms:
                accessible_obj_ids.append(obj_acl.obj_id)

        return accessible_obj_ids

__init__(collection_name, session_key, app, owner='nobody', scheme=None, host=None, port=None, **context)

Initializes ObjectACLManager.

Parameters:

Name Type Description Default
collection_name str

Collection name to store object ACL info.

 required
session_key str

Splunk access token.

 required
app str

App name of namespace.

 required
owner Optional[str]

(optional) Owner of namespace, default is nobody.

 'nobody'
scheme Optional[str]

(optional) The access scheme, default is None.

 None
host Optional[str]

(optional) The host name, default is None.

 None
port Optional[int]

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}

Raises:

Type Description
ObjectACLManagerException

If init ObjectACLManager failed.
Source code in solnlib/user_access.py 
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
def __init__(
    self,
    collection_name: str,
    session_key: str,
    app: str,
    owner: Optional[str] = "nobody",
    scheme: Optional[str] = None,
    host: Optional[str] = None,
    port: Optional[int] = None,
    **context: dict,
):
    """Initializes ObjectACLManager.

    Arguments:
        collection_name: Collection name to store object ACL info.
        session_key: Splunk access token.
        app: App name of namespace.
        owner: (optional) Owner of namespace, default is `nobody`.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Raises:
        ObjectACLManagerException: If init ObjectACLManager failed.
    """
    collection_name = "{app}_{collection_name}".format(
        app=app, collection_name=collection_name
    )
    try:
        self._collection_data = _utils.get_collection_data(
            collection_name,
            session_key,
            app,
            owner,
            scheme,
            host,
            port,
            None,
            **context,
        )
    except KeyError:
        raise ObjectACLManagerException(
            f"Get object acl collection: {collection_name} fail."
        )

delete_acl(obj_collection, obj_id)

Delete acl info.

Query object acl info with parameter of the combination of obj_collection and obj_ids from KVStore and delete it.

Parameters:

Name Type Description Default
obj_collection str

Collection where object currently stored.

 required
obj_id str

ID of this object.

 required

Raises:

Type Description
ObjectACLNotExistException

If object ACL info does not exist.
Source code in solnlib/user_access.py 
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
@utils.retry(exceptions=[binding.HTTPError])
def delete_acl(self, obj_collection: str, obj_id: str):
    """Delete acl info.

    Query object acl info with parameter of the combination of
    `obj_collection` and `obj_ids` from KVStore and delete it.

    Arguments:
        obj_collection: Collection where object currently stored.
        obj_id: ID of this object.

    Raises:
        ObjectACLNotExistException: If object ACL info does not exist.
    """

    key = ObjectACL.generate_key(obj_collection, obj_id)
    try:
        self._collection_data.delete_by_id(key)
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise ObjectACLNotExistException(
            "Object ACL info of {}_{} does not exist.".format(
                obj_collection, obj_id
            )
        )

delete_acls(obj_collection, obj_ids)

Batch delete acl info.

Query objects acl info with parameter of the combination of obj_collection and obj_ids from KVStore and delete them.

Parameters:

Name Type Description Default
obj_collection str

Collection where object currently stored.

 required
obj_ids List[str]

IDs of objects.

 required
Source code in solnlib/user_access.py 
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
@utils.retry(exceptions=[binding.HTTPError])
def delete_acls(self, obj_collection: str, obj_ids: List[str]):
    """Batch delete acl info.

    Query objects acl info with parameter of the combination of
    `obj_collection` and `obj_ids` from KVStore and delete them.

    Arguments:
        obj_collection: Collection where object currently stored.
        obj_ids: IDs of objects.
    """

    query = json.dumps(
        {
            "$or": [
                {"_key": ObjectACL.generate_key(obj_collection, obj_id)}
                for obj_id in obj_ids
            ]
        }
    )
    self._collection_data.delete(query=query)

get_accessible_object_ids(user, operation, obj_collection, obj_ids)

Get accessible IDs of objects from obj_acls.

Parameters:

Name Type Description Default
user str

User name of current operation.

 required
operation str

User operation, possible option: (read/write/delete).

 required
obj_collection str

Collection where object currently stored.

 required
obj_ids List[str]

IDs of objects.

 required

Returns:

Type Description
List[str]

List of IDs of accessible objects.
Source code in solnlib/user_access.py 
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
@utils.retry(exceptions=[binding.HTTPError])
def get_accessible_object_ids(
    self, user: str, operation: str, obj_collection: str, obj_ids: List[str]
) -> List[str]:
    """Get accessible IDs of objects from `obj_acls`.

    Arguments:
        user: User name of current `operation`.
        operation: User operation, possible option: (read/write/delete).
        obj_collection: Collection where object currently stored.
        obj_ids: IDs of objects.

    Returns:
        List of IDs of accessible objects.
    """

    obj_acls = self.get_acls(obj_collection, obj_ids)
    accessible_obj_ids = []
    for obj_acl in obj_acls:
        perms = obj_acl.obj_perms[operation]
        if ObjectACL.OBJ_PERMS_ALLOW_ALL in perms or user in perms:
            accessible_obj_ids.append(obj_acl.obj_id)

    return accessible_obj_ids

get_acl(obj_collection, obj_id)

Get acl info.

Query object acl info with parameter of the combination of obj_collection and obj_id from self.collection_name and return it.

Parameters:

Name Type Description Default
obj_collection str

Collection where object currently stored.

 required
obj_id str

ID of this object.

 required

Returns:

Type Description
ObjectACL

Object acl info if success else None.

Raises:

Type Description
ObjectACLNotExistException

If object ACL info does not exist.
Source code in solnlib/user_access.py 
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
@utils.retry(exceptions=[binding.HTTPError])
def get_acl(self, obj_collection: str, obj_id: str) -> "ObjectACL":
    """Get acl info.

    Query object acl info with parameter of the combination of
    `obj_collection` and `obj_id` from `self.collection_name` and
    return it.

    Arguments:
        obj_collection: Collection where object currently stored.
        obj_id: ID of this object.

    Returns:
        Object acl info if success else None.

    Raises:
        ObjectACLNotExistException: If object ACL info does not exist.
    """

    key = ObjectACL.generate_key(obj_collection, obj_id)
    try:
        obj_acl = self._collection_data.query_by_id(key)
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise ObjectACLNotExistException(
            "Object ACL info of {}_{} does not exist.".format(
                obj_collection, obj_id
            )
        )

    return ObjectACL.parse(obj_acl)

get_acls(obj_collection, obj_ids)

Batch get acl info.

Query objects acl info with parameter of the combination of obj_collection and obj_ids from KVStore and return them.

Parameters:

Name Type Description Default
obj_collection str

Collection where object currently stored.

 required
obj_ids List[str]

IDs of objects.

 required

Returns:

Type Description
List[ObjectACL]

List of ObjectACL instances.
Source code in solnlib/user_access.py 
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
@utils.retry(exceptions=[binding.HTTPError])
def get_acls(self, obj_collection: str, obj_ids: List[str]) -> List[ObjectACL]:
    """Batch get acl info.

    Query objects acl info with parameter of the combination of
    `obj_collection` and `obj_ids` from KVStore and return them.

    Arguments:
        obj_collection: Collection where object currently stored.
        obj_ids: IDs of objects.

    Returns:
        List of `ObjectACL` instances.
    """

    query = json.dumps(
        {
            "$or": [
                {"_key": ObjectACL.generate_key(obj_collection, obj_id)}
                for obj_id in obj_ids
            ]
        }
    )
    obj_acls = self._collection_data.query(query=query)

    return [ObjectACL.parse(obj_acl) for obj_acl in obj_acls]

update_acl(obj_collection, obj_id, obj_type, obj_app, obj_owner, obj_perms, obj_shared_by_inclusion=True, replace_existing=True)

Update acl info of object.

Construct a new object acl info first, if replace_existing is True then replace existing acl info else merge new object acl info with the old one and replace the old acl info with merged acl info.

Parameters:

Name Type Description Default
obj_collection str

Collection where object currently stored.

 required
obj_id str

ID of this object.

 required
obj_type str

Type of this object.

 required
obj_app str

App of this object.

 required
obj_owner str

Owner of this object.

 required
obj_perms dict

Object perms, like:

{ ‘read’: [‘*’], ‘write’: [‘admin’], ‘delete’: [‘admin’] }.

 required
obj_shared_by_inclusion bool

(optional) Flag of object is shared by inclusion, default is True.

 True
replace_existing bool

(optional) Replace existing acl info flag, True indicates replace old acl info with new one else merge with old acl info, default is True.

 True
Source code in solnlib/user_access.py 
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
@utils.retry(exceptions=[binding.HTTPError])
def update_acl(
    self,
    obj_collection: str,
    obj_id: str,
    obj_type: str,
    obj_app: str,
    obj_owner: str,
    obj_perms: dict,
    obj_shared_by_inclusion: bool = True,
    replace_existing: bool = True,
):
    """Update acl info of object.

    Construct a new object acl info first, if `replace_existing` is True
    then replace existing acl info else merge new object acl info with the
    old one and replace the old acl info with merged acl info.

    Arguments:
        obj_collection: Collection where object currently stored.
        obj_id: ID of this object.
        obj_type: Type of this object.
        obj_app: App of this object.
        obj_owner: Owner of this object.
        obj_perms: Object perms, like:

            {
                'read': ['*'],
                'write': ['admin'],
                'delete': ['admin']
            }.
        obj_shared_by_inclusion: (optional) Flag of object is shared by
            inclusion, default is True.
        replace_existing: (optional) Replace existing acl info flag, True
            indicates replace old acl info with new one else merge with old
            acl info, default is True.
    """

    obj_acl = ObjectACL(
        obj_collection,
        obj_id,
        obj_type,
        obj_app,
        obj_owner,
        obj_perms,
        obj_shared_by_inclusion,
    )

    if not replace_existing:
        try:
            old_obj_acl = self.get_acl(obj_collection, obj_id)
        except ObjectACLNotExistException:
            old_obj_acl = None

        if old_obj_acl:
            obj_acl.merge(old_obj_acl)

    self._collection_data.batch_save(obj_acl.record)

update_acls(obj_collection, obj_ids, obj_type, obj_app, obj_owner, obj_perms, obj_shared_by_inclusion=True, replace_existing=True)

Batch update object acl info to all provided obj_ids.

Parameters:

Name Type Description Default
obj_collection str

Collection where objects currently stored.

 required
obj_ids List[str]

IDs list of objects.

 required
obj_type str

Type of this object.

 required
obj_app str

App of this object.

 required
obj_owner str

Owner of this object.

 required
obj_perms dict

Object perms, like:

{ ‘read’: [‘*’], ‘write’: [‘admin’], ‘delete’: [‘admin’] }.

 required
obj_shared_by_inclusion bool

(optional) Flag of object is shared by inclusion, default is True.

 True
replace_existing bool

(optional) Replace existing acl info flag, True indicates replace old acl info with new one else merge with old acl info, default is True.

 True
Source code in solnlib/user_access.py 
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
@utils.retry(exceptions=[binding.HTTPError])
def update_acls(
    self,
    obj_collection: str,
    obj_ids: List[str],
    obj_type: str,
    obj_app: str,
    obj_owner: str,
    obj_perms: dict,
    obj_shared_by_inclusion: bool = True,
    replace_existing: bool = True,
):
    """Batch update object acl info to all provided `obj_ids`.

    Arguments:
        obj_collection: Collection where objects currently stored.
        obj_ids: IDs list of objects.
        obj_type: Type of this object.
        obj_app: App of this object.
        obj_owner: Owner of this object.
        obj_perms: Object perms, like:

            {
                'read': ['*'],
                'write': ['admin'],
                'delete': ['admin']
            }.
        obj_shared_by_inclusion: (optional) Flag of object is shared by
            inclusion, default is True.
        replace_existing: (optional) Replace existing acl info flag, True
            indicates replace old acl info with new one else merge with old acl
            info, default is True.
    """

    obj_acl_records = []
    for obj_id in obj_ids:
        obj_acl = ObjectACL(
            obj_collection,
            obj_id,
            obj_type,
            obj_app,
            obj_owner,
            obj_perms,
            obj_shared_by_inclusion,
        )

        if not replace_existing:
            try:
                old_obj_acl = self.get_acl(obj_collection, obj_id)
            except ObjectACLNotExistException:
                old_obj_acl = None

            if old_obj_acl:
                obj_acl.merge(old_obj_acl)

        obj_acl_records.append(obj_acl.record)

    self._collection_data.batch_save(*obj_acl_records)

ObjectACLManagerException

Bases: Exception

Exception for ObjectACLManager.

Source code in solnlib/user_access.py 
220
221
222
223
class ObjectACLManagerException(Exception):
    """Exception for ObjectACLManager."""

    pass

ObjectACLNotExistException

Bases: Exception

Exception for the situation when ACL does not exist.

Source code in solnlib/user_access.py 
226
227
228
229
class ObjectACLNotExistException(Exception):
    """Exception for the situation when ACL does not exist."""

    pass

UserAccessException

Bases: Exception

Exception for the situation when there is user access exception.

Source code in solnlib/user_access.py 
697
698
699
700
class UserAccessException(Exception):
    """Exception for the situation when there is user access exception."""

    pass

UserNotExistException

Bases: Exception

Exception when user does not exist.

Source code in solnlib/user_access.py 
823
824
825
826
class UserNotExistException(Exception):
    """Exception when user does not exist."""

    pass

check_user_access(session_key, capabilities, obj_type, operation, scheme=None, host=None, port=None, **context)

User access checker.

It will fetch user capabilities from given session_key and check if the capability extracted from capabilities, obj_type and operation is contained, if user capabilities include the extracted capability user access is ok else fail.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
capabilities dict

App capabilities, example:

{ ‘object_type1’: { ‘read’: ‘read_app_object_type1’, ‘write’: ‘write_app_object_type1’, ‘delete’: ‘delete_app_object_type1’}, ‘object_type2’: { ‘read’: ‘read_app_object_type2’, ‘write’: ‘write_app_object_type2’, ‘delete’: ‘delete_app_object_type2’ }, … }

 required
obj_type str

Object type.

 required
operation str

User operation, possible option: (read/write/delete).

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}

Raises:

Type Description
UserAccessException

If user access permission is denied.

Examples:

>>> from solnlib.user_access import check_user_access
>>> def fun():
>>>     check_user_access(
>>>         session_key, capabilities, 'test_object', 'read')
>>>     ...
Source code in solnlib/user_access.py 
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
def check_user_access(
    session_key: str,
    capabilities: dict,
    obj_type: str,
    operation: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
):
    """User access checker.

    It will fetch user capabilities from given `session_key` and check if
    the capability extracted from `capabilities`, `obj_type` and `operation`
    is contained, if user capabilities include the extracted capability user
    access is ok else fail.

    Arguments:
        session_key: Splunk access token.
        capabilities: App capabilities, example:

            {
                'object_type1': {
                    'read': 'read_app_object_type1',
                    'write': 'write_app_object_type1',
                    'delete': 'delete_app_object_type1'},
                    'object_type2': {
                    'read': 'read_app_object_type2',
                    'write': 'write_app_object_type2',
                    'delete': 'delete_app_object_type2'
                },
                ...
            }
        obj_type: Object type.
        operation: User operation, possible option: (read/write/delete).
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Raises:
        UserAccessException: If user access permission is denied.

    Examples:
       >>> from solnlib.user_access import check_user_access
       >>> def fun():
       >>>     check_user_access(
       >>>         session_key, capabilities, 'test_object', 'read')
       >>>     ...
    """

    username = get_current_username(
        session_key, scheme=scheme, host=host, port=port, **context
    )
    capability = capabilities[obj_type][operation]
    if not user_is_capable(
        session_key,
        username,
        capability,
        scheme=scheme,
        host=host,
        port=port,
        **context,
    ):
        raise UserAccessException(
            "Permission denied, %s does not have the capability: %s."
            % (username, capability)
        )

get_current_username(session_key, scheme=None, host=None, port=None, **context)

Get current user name from session_key.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}

Returns:

Type Description
str

Current user name.

Raises:

Type Description
InvalidSessionKeyException

If session_key is invalid.

Examples:

>>> from solnlib import user_access
>>> user_name = user_access.get_current_username(session_key)
Source code in solnlib/user_access.py 
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
@utils.retry(exceptions=[binding.HTTPError])
def get_current_username(
    session_key: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
) -> str:
    """Get current user name from `session_key`.

    Arguments:
        session_key: Splunk access token.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Returns:
        Current user name.

    Raises:
        InvalidSessionKeyException: If `session_key` is invalid.

    Examples:
       >>> from solnlib import user_access
       >>> user_name = user_access.get_current_username(session_key)
    """

    _rest_client = rest_client.SplunkRestClient(
        session_key, "-", scheme=scheme, host=host, port=port, **context
    )
    try:
        response = _rest_client.get(
            "/services/authentication/current-context", output_mode="json"
        ).body.read()
    except binding.HTTPError as e:
        if e.status != 401:
            raise

        raise InvalidSessionKeyException("Invalid session key.")

    return json.loads(response)["entry"][0]["content"]["username"]

get_user_capabilities(session_key, username, scheme=None, host=None, port=None, **context)

Get user capabilities.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}

Returns:

Type Description
List[dict]

User capabilities.

Raises:

Type Description
UserNotExistException

If username does not exist.

Examples:

>>> from solnlib import user_access
>>> user_capabilities = user_access.get_user_capabilities(
>>>     session_key, 'test_user')
Source code in solnlib/user_access.py 
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
@utils.retry(exceptions=[binding.HTTPError])
def get_user_capabilities(
    session_key: str,
    username: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
) -> List[dict]:
    """Get user capabilities.

    Arguments:
        session_key: Splunk access token.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Returns:
        User capabilities.

    Raises:
        UserNotExistException: If `username` does not exist.

    Examples:
       >>> from solnlib import user_access
       >>> user_capabilities = user_access.get_user_capabilities(
       >>>     session_key, 'test_user')
    """

    _rest_client = rest_client.SplunkRestClient(
        session_key, "-", scheme=scheme, host=host, port=port, **context
    )
    url = f"/services/authentication/users/{username}"
    try:
        response = _rest_client.get(url, output_mode="json").body.read()
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise UserNotExistException("User: %s does not exist." % username)

    return json.loads(response)["entry"][0]["content"]["capabilities"]

get_user_roles(session_key, username, scheme=None, host=None, port=None, **context)

Get user roles.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
username str

(optional) User name of roles to get.

 required
scheme

(optional) The access scheme, default is None.

 None
host

(optional) The host name, default is None.

 None
port

(optional) The port number, default is None.

 None
context

Other configurations for Splunk rest client.

 {}

Returns:

Type Description
List

User roles.

Raises:

Type Description
UserNotExistException

If username does not exist.

Examples:

>>> from solnlib import user_access
>>> user_roles = user_access.get_user_roles(session_key, 'test_user')
Source code in solnlib/user_access.py 
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
@utils.retry(exceptions=[binding.HTTPError])
def get_user_roles(
    session_key: str, username: str, scheme=None, host=None, port=None, **context
) -> List:
    """Get user roles.

    Arguments:
        session_key: Splunk access token.
        username: (optional) User name of roles to get.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Returns:
        User roles.

    Raises:
        UserNotExistException: If `username` does not exist.

    Examples:
       >>> from solnlib import user_access
       >>> user_roles = user_access.get_user_roles(session_key, 'test_user')
    """

    _rest_client = rest_client.SplunkRestClient(
        session_key, "-", scheme=scheme, host=host, port=port, **context
    )
    url = f"/services/authentication/users/{username}"
    try:
        response = _rest_client.get(url, output_mode="json").body.read()
    except binding.HTTPError as e:
        if e.status != 404:
            raise

        raise UserNotExistException("User: %s does not exist." % username)

    return json.loads(response)["entry"][0]["content"]["roles"]

user_is_capable(session_key, username, capability, scheme=None, host=None, port=None, **context)

Check if user is capable for given capability.

Parameters:

Name Type Description Default
session_key str

Splunk access token.

 required
username str

(optional) User name of roles to get.

 required
capability str

The capability we wish to check for.

 required
scheme str

(optional) The access scheme, default is None.

 None
host str

(optional) The host name, default is None.

 None
port int

(optional) The port number, default is None.

 None
context dict

Other configurations for Splunk rest client.

 {}

Returns:

Type Description
bool

True if user is capable else False.

Raises:

Type Description
UserNotExistException

If username does not exist.

Examples:

>>> from solnlib import user_access
>>> is_capable = user_access.user_is_capable(
>>>     session_key, 'test_user', 'object_read_capability')
Source code in solnlib/user_access.py 
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
def user_is_capable(
    session_key: str,
    username: str,
    capability: str,
    scheme: str = None,
    host: str = None,
    port: int = None,
    **context: dict,
) -> bool:
    """Check if user is capable for given `capability`.

    Arguments:
        session_key: Splunk access token.
        username: (optional) User name of roles to get.
        capability: The capability we wish to check for.
        scheme: (optional) The access scheme, default is None.
        host: (optional) The host name, default is None.
        port: (optional) The port number, default is None.
        context: Other configurations for Splunk rest client.

    Returns:
        True if user is capable else False.

    Raises:
        UserNotExistException: If `username` does not exist.

    Examples:
       >>> from solnlib import user_access
       >>> is_capable = user_access.user_is_capable(
       >>>     session_key, 'test_user', 'object_read_capability')
    """

    capabilities = get_user_capabilities(
        session_key, username, scheme=scheme, host=host, port=port, **context
    )
    return capability in capabilities

utils.py

Common utilities.

__all__ = ['handle_teardown_signals', 'datetime_to_seconds', 'is_true', 'is_false', 'retry', 'extract_http_scheme_host_port', 'remove_http_proxy_env_vars'] module-attribute

datetime_to_seconds(dt)

Convert UTC datetime to seconds since epoch.

Parameters:

Name Type Description Default
dt datetime.datetime

Date time.

 required

Returns:

Type Description
float

Seconds since epoch.
Source code in solnlib/utils.py 
82
83
84
85
86
87
88
89
90
91
92
93
def datetime_to_seconds(dt: datetime.datetime) -> float:
    """Convert UTC datetime to seconds since epoch.

    Arguments:
        dt: Date time.

    Returns:
        Seconds since epoch.
    """

    epoch_time = datetime.datetime.utcfromtimestamp(0)
    return (dt - epoch_time).total_seconds()

extract_http_scheme_host_port(http_url)

Extract scheme, host and port from a HTTP URL.

Parameters:

Name Type Description Default
http_url str

HTTP URL to extract.

 required

Returns:

Type Description
Tuple

A tuple of scheme, host and port

Raises:

Type Description
ValueError

If http_url is not in http(s)://hostname:port format.
Source code in solnlib/utils.py 
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
def extract_http_scheme_host_port(http_url: str) -> Tuple:
    """Extract scheme, host and port from a HTTP URL.

    Arguments:
        http_url: HTTP URL to extract.

    Returns:
        A tuple of scheme, host and port

    Raises:
        ValueError: If `http_url` is not in http(s)://hostname:port format.
    """

    http_info = urlparse.urlparse(http_url)
    if not http_info.scheme or not http_info.hostname or not http_info.port:
        raise ValueError(http_url + " is not in http(s)://hostname:port format")
    return http_info.scheme, http_info.hostname, http_info.port

get_appname_from_path(absolute_path)

Gets name of the app from its path.

Parameters:

Name Type Description Default
absolute_path

path of app

 required
Source code in solnlib/utils.py 
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
def get_appname_from_path(absolute_path):
    """Gets name of the app from its path.

    Arguments:
        absolute_path: path of app

    Returns:
    """
    absolute_path = os.path.normpath(absolute_path)
    parts = absolute_path.split(os.path.sep)
    parts.reverse()
    for key in ("apps", "peer-apps", "manager-apps"):
        try:
            idx = parts.index(key)
        except ValueError:
            continue
        else:
            try:
                if parts[idx + 1] == "etc":
                    return parts[idx - 1]
            except IndexError:
                pass
            continue
    return "-"

handle_teardown_signals(callback)

Register handler for SIGTERM/SIGINT/SIGBREAK signal.

Catch SIGTERM/SIGINT/SIGBREAK signals, and invoke callback Note: this should be called in main thread since Python only catches signals in main thread.

Parameters:

Name Type Description Default
callback Callable

Callback for tear down signals.

 required
Source code in solnlib/utils.py 
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
def handle_teardown_signals(callback: Callable):
    """Register handler for SIGTERM/SIGINT/SIGBREAK signal.

    Catch SIGTERM/SIGINT/SIGBREAK signals, and invoke callback
    Note: this should be called in main thread since Python only catches
    signals in main thread.

    Arguments:
        callback: Callback for tear down signals.
    """

    signal.signal(signal.SIGTERM, callback)
    signal.signal(signal.SIGINT, callback)

    if os.name == "nt":
        signal.signal(signal.SIGBREAK, callback)

is_false(val)

Decide if val is false.

Parameters:

Name Type Description Default
val Union[str, int]

Value to check.

 required

Returns:

Type Description
bool

True or False.
Source code in solnlib/utils.py 
112
113
114
115
116
117
118
119
120
121
122
123
124
125
def is_false(val: Union[str, int]) -> bool:
    """Decide if `val` is false.

    Arguments:
        val: Value to check.

    Returns:
        True or False.
    """

    value = str(val).strip().upper()
    if value in ("0", "FALSE", "F", "N", "NO", "NONE", ""):
        return True
    return False

is_true(val)

Decide if val is true.

Parameters:

Name Type Description Default
val Union[str, int]

Value to check.

 required

Returns:

Type Description
bool

True or False.
Source code in solnlib/utils.py 
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
def is_true(val: Union[str, int]) -> bool:
    """Decide if `val` is true.

    Arguments:
        val: Value to check.

    Returns:
        True or False.
    """

    value = str(val).strip().upper()
    if value in ("1", "TRUE", "T", "Y", "YES"):
        return True
    return False

remove_http_proxy_env_vars()

Removes HTTP(s) proxies from environment variables.

Removes the following environment variables
  • http_proxy
  • https_proxy
  • HTTP_PROXY
  • HTTPS_PROXY

This function can be used in Splunk modular inputs code before starting the ingestion to ensure that no proxy is going to be used when doing requests. In case of proxy is needed, it can be defined in the modular inputs code.

Source code in solnlib/utils.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def remove_http_proxy_env_vars() -> None:
    """Removes HTTP(s) proxies from environment variables.

    Removes the following environment variables:
        * http_proxy
        * https_proxy
        * HTTP_PROXY
        * HTTPS_PROXY

    This function can be used in Splunk modular inputs code before starting the
    ingestion to ensure that no proxy is going to be used when doing requests.
    In case of proxy is needed, it can be defined in the modular inputs code.
    """
    env_vars_to_remove = (
        "http_proxy",
        "https_proxy",
        "HTTP_PROXY",
        "HTTPS_PROXY",
    )
    for env_var in env_vars_to_remove:
        if env_var in os.environ:
            del os.environ[env_var]

retry(retries=3, reraise=True, default_return=None, exceptions=None)

A decorator to run function with max retries times if there is exception.

Parameters:

Name Type Description Default
retries int

(optional) Max retries times, default is 3.

 3
reraise bool

Whether exception should be reraised, default is True.

 True
default_return Any

(optional) Default return value for function run after max retries and reraise is False.

 None
exceptions List

(optional) List of exceptions that should retry.

 None
Source code in solnlib/utils.py 
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
def retry(
    retries: int = 3,
    reraise: bool = True,
    default_return: Any = None,
    exceptions: List = None,
):
    """A decorator to run function with max `retries` times if there is
    exception.

    Arguments:
        retries: (optional) Max retries times, default is 3.
        reraise: Whether exception should be reraised, default is True.
        default_return: (optional) Default return value for function
            run after max retries and reraise is False.
        exceptions: (optional) List of exceptions that should retry.
    """

    max_tries = max(retries, 0) + 1

    def do_retry(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            last_ex = None
            for i in range(max_tries):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    logging.warning(
                        "Run function: %s failed: %s.",
                        func.__name__,
                        traceback.format_exc(),
                    )
                    if not exceptions or any(
                        isinstance(e, exception) for exception in exceptions
                    ):
                        last_ex = e
                        if i < max_tries - 1:
                            time.sleep(2**i)
                    else:
                        raise

            if reraise:
                raise last_ex
            else:
                return default_return

        return wrapper

    return do_retry

Ended: References