Creating a set of resources for a storage

Managing Kaspersky Unified Monitoring and Analysis Platform > Administrator's guide > KUMA services > Creating a storage > Creating a set of resources for a storage

Creating a set of resources for a storage

Expand all | Collapse all

In the KUMA Console, a storage service is created based on the set of resources for the storage.

To create a set of resources for a storage in the KUMA Console:

In the KUMA Console, under Resources → Storages, click Add storage.
This opens the Create storage window.
On the Basic settings tab, in the Storage name field, enter a unique name for the service you are creating. The name must contain 1 to 128 Unicode characters.
In the Tenant drop-down list, select the tenant that will own the storage.
In the Tags drop-down list, select the tags for the resource set that you are creating.
The list includes all available tags created in the tenant of the resource and in the Shared tenant. You can find a tag in the list by typing its name in the field. If the tag you entered does not exist, you can press Enter or click Add to create it.
You can optionally add up to 256 Unicode characters describing the service in the Description field.
In the Storage condition options field, select an event storage condition in the ClickHouse cluster for the storage, which, when satisfied, will cause events to be transferred to cold storage disks or deleted if cold storage is not configured or is configured incorrectly. The condition is applied to the default space and to events from deleted spaces.
By default, ClickHouse moves events to cold storage disks or deletes them if more than 97% of the storage is full. KUMA also applies an additional 365 days storage condition when creating a storage. You can configure custom storage conditions for more stable performance of the storage.

To set the storage condition, do one of the following:
- If you want to limit the storage period for events, select Days from the drop-down list, and in the field, specify the maximum event storage period (in days) in the ClickHouse hot storage cluster.
  After the specified period, events are automatically transferred to cold storage disks or deleted from the ClickHouse cluster, starting with the partitions with the oldest date. The minimum value is 1. The default value is 365.
- If you want to limit the maximum storage size, select GB from the drop-down list, and in the field, specify the maximum storage size in gigabytes.
  When the storage reaches the specified size, events are automatically transferred to cold storage disks or deleted from the ClickHouse cluster, starting with the partitions with the oldest date. The minimum value and default value is 1.
- If you want to limit the storage size to a percentage of disk space that is available to the storage (according to VictoriaMetrics), select Percentage from the drop-down list, and in the field, specify the maximum storage size as a percentage of the available disk space. In this case, the condition can also be triggered when the disk space available to the storage is decreased.
  When the storage reaches the specified percentage of disk space available to it, events are automatically transferred to cold storage disks or deleted from the ClickHouse cluster, starting with the partitions with the oldest date. Possible values: 1 to 95. The default value is 80. If you want to use percentages for all storage spaces, the sum total of percentages in the conditions of all spaces may not exceed 95, but we recommend specifying a limit of at most 90% for the entire storage or for individual spaces.
  
  We do not recommend specifying small percentage values because this increases the probability of data loss in the storage.
For [OOTV] Storage, the default event storage period is 2 days. If you want to use this storage, you can change the event storage condition for it, if necessary.
If you want to use an additional storage condition, click Add storage condition and specify an additional storage condition as described in step 6.
The maximum number of conditions is two, and you can combine only conditions the following types:
- Days and storage size in GB
- Days and storage size as a percentage
If you want to delete a storage condition, click the X icon next to this condition.
In the Audit retention period field, specify the period, in days, to store audit events. The minimum value and default value is 365.
If cold storage is required, specify the event storage term:
- Event retention time specifies the total KUMA event storage duration in days, counting from the moment when the event is received. When the specified period expires, events are automatically deleted from the cold storage disk. The default value is 0.
  The event retention time is calculated as the sum of the event retention time in the ClickHouse hot storage cluster until the condition specified in the Storage condition options setting is triggered, and the event retention time on the cold storage disk. After one of storage conditions is triggered, the data partition for the earliest date is moved to the cold storage disk, and there it remains until the event retention time in KUMA expires.
  
  Depending on the specified storage condition, the resulting retention time is as follows:
  - If you specified a storage condition in days, the Event retention time must be strictly greater than the number of days specified in the storage condition. You can calculate the cold storage duration for events as the Event retention time minus the number of days specified in the Storage condition options setting.
    If you do not want to store events on the cold storage disk, you can specify the same number of days in the Event retention time field as in the storage condition.
  - If you specified the storage condition in terms of disk size (absolute or percentage), the minimum value of the Event retention time is 1. The cold storage duration for events is calculated as Event retention time minus the number of days from the receipt of the event to triggering of the condition and the disk partition filling up, but until the condition is triggered, calculating an exact duration is impossible. In this case, we recommend specifying a relatively large value for Event retention time to avoid events being deleted.
    If you do not want to store events on the cold storage disk, you can set Event retention time to 0.
- Audit cold retention period—the number of days to store audit events. The minimum value is 0.
The Event retention time and Audit cold retention period settings become available only after at least one cold storage disk has been added.
If you want to change ClickHouse settings, in the ClickHouse configuration override field, paste the lines with settings from the ClickHouse configuration XML file /opt/kaspersky/kuma/clickhouse/cfg/config.xml. Specifying the root elements <yandex>, </yandex> is not required. Settings passed in this field are used instead of the default settings.
Example:

<merge_tree>

<parts_to_delay_insert>600</parts_to_delay_insert>

<parts_to_throw_insert>1100</parts_to_throw_insert>

</merge_tree>
Use the Debug toggle switch to specify whether resource logging must be enabled. If you want to only log errors for all KUMA components, disable debugging. If you want to get detailed information in the logs, enable debugging.
If necessary, in the ClickHouse cluster nodes section, add ClickHouse cluster nodes to the storage.
There can be multiple nodes. You can add nodes by clicking the Add node button or remove nodes by clicking the X icon of the relevant node.

Available settings:
- In the FQDN field, enter the fully qualified domain name of the node that you want to add. For example, kuma-storage-cluster1-server1.example.com.
- In the Shard ID, Replica ID, and Keeper ID fields, specify the role of the node in the ClickHouse cluster. The shard and keeper IDs must be unique within the cluster, the replica ID must be unique within the shard. The following example shows how to populate the ClickHouse cluster nodes section for a storage with dedicated keepers in a distributed installation. You can adapt the example to suit your needs.
  Example:
  
  ClickHouse cluster nodes
  
  FQDN: kuma-storage-cluster1-server1.example.com
  
  Shard ID: 0
  
  Replica ID: 0
  
  Keeper ID: 1
  
  FQDN: kuma-storage-cluster1server2.example.com
  
  Shard ID: 0
  
  Replica ID: 0
  
  Keeper ID: 2
  
  FQDN: kuma-storage-cluster1server3.example.com
  
  Shard ID: 0
  
  Replica ID: 0
  
  Keeper ID: 3
  
  FQDN: kuma-storage-cluster1server4.example.com
  
  Shard ID: 1
  
  Replica ID: 1
  
  Keeper ID: 0
  
  FQDN: kuma-storage-cluster1server5.example.com
  
  Shard ID: 1
  
  Replica ID: 2
  
  Keeper ID: 0
  
  FQDN: kuma-storage-cluster1server6.example.com
  
  Shard ID: 2
  
  Replica ID: 1
  
  Keeper ID: 0
  
  FQDN: kuma-storage-cluster1server7.example.com
  
  Shard ID: 2
  
  Replica ID: 2
  
  Keeper ID: 0
  
  Distributed Installation diagram
If necessary, in the Spaces section, add spaces to the storage to distribute the stored events.
There can be multiple spaces. You can add spaces by clicking the Add space button or remove spaces by clicking the X icon of the relevant space.

Available settings:
- In the Name field, specify a name for the space containing 1 to 128 Unicode characters.
- In the Storage condition options field, select an event storage condition in the ClickHouse cluster for the space, which, when satisfied, will cause events to be transferred to cold storage disks or deleted if cold storage is not configured or is configured incorrectly. KUMA applies the 365 days storage condition when a space is added.
  To set the storage condition for a space, do one of the following:
  - If you want to limit the storage period for events, select Days from the drop-down list, and in the field, specify the maximum event storage period (in days) in the ClickHouse hot storage cluster.
    After the specified period, events are automatically transferred to cold storage disks or deleted from the ClickHouse cluster, starting with the partitions with the oldest date. The minimum value is 1. The default value is 365.
  - If you want to limit the maximum storage space size, select GB from the drop-down list, and in the field, specify the maximum space size in gigabytes.
    When the space reaches the specified size, events are automatically transferred to cold storage disks or deleted from the ClickHouse cluster, starting with the partitions with the oldest date. The minimum value and default value is 1.
  - If you want to limit the space size to a percentage of disk space that is available to the storage (according to VictoriaMetrics), select Percentage from the drop-down list, and in the field, specify the maximum space size as a percentage of the size of the disk available to the storage. In this case, the condition can also be triggered when the disk space available to the storage is decreased.
    When the space reaches the specified percentage of disk space available to the storage, events are automatically transferred to cold storage disks or deleted from the ClickHouse cluster, starting with the partitions with the oldest date. Possible values: 1 to 95. The default value is 80. If you want to use percentages for all storage spaces, the sum total of percentages in the conditions of all spaces may not exceed 95, but we recommend specifying a limit of at most 90% for the entire storage or for individual spaces.
    
    We do not recommend specifying small percentage values because this increases the probability of data loss in the storage.
  When using size as the storage condition, you must ensure that the total size of all spaces specified in the storage conditions does not exceed the physical size of the storage, otherwise an error will be displayed when starting the service.
  
  In storage conditions with a size limitation, use the same units of measure for all spaces of a storage (only gigabytes or only percentage values). Otherwise, if the condition is specified as a percentage for one space, and in gigabytes for another space, the storage may overflow due to mismatch of values, leading to data loss.
- If you want to make a space inactive if it is outdated and no longer relevant, select the Read only check box.
  This prevents events from going into that space. To make the space active again, clear the Read only check box. This check box is cleared by default.
- If necessary, in the Event retention time field, specify the total KUMA event storage duration in days, counting from the moment when the event is received. When the specified period expires, events are automatically deleted from the cold storage disk. The default value is 0.
  The event retention time is calculated as the sum of the event retention time in the ClickHouse hot storage cluster until the condition specified in the Storage condition options setting is triggered, and the event retention time on the cold storage disk. After one of storage conditions is triggered, the data partition for the earliest date is moved to the cold storage disk, and there it remains until the event retention time in KUMA expires.
  
  Depending on the specified storage condition, the resulting retention time is as follows:
  - If you specified a storage condition in days, the Event retention time must be strictly greater than the number of days specified in the storage condition. The cold storage duration for events is calculated as the Event retention time minus the number of days specified in the Storage condition options setting.
    If you do not want to store events from this space on the cold storage disk, you can specify the same number of days in the Event retention time field as in the storage condition.
  - If you specified the storage condition in terms of disk size (absolute or percentage), the minimum value of the Event retention time is 1. The cold storage duration for events is calculated as Event retention time minus the number of days from the receipt of the event to triggering of the condition and the disk partition filling up, but until the condition is triggered, calculating an exact duration is impossible. In this case, we recommend specifying a relatively large value for Event retention time to avoid events being deleted.
    If you do not want to store events from this space on the cold storage disk, you can set Event retention time to 0.
  The Event retention time setting becomes available only after adding at least one cold storage disk.
- In the Filter settings section, you can specify conditions to identify events that will be put into this space. To create a new filter, in the Filter drop-down list, select an existing filter or Create new.
  Creating a filter in resources
  To create a filter:
  1. In the Filter drop-down list, select Create new.
  2. If you want to keep the filter as a separate resource, select the Save filter check box. In this case, you will be able to use the created filter in various services. This check box is cleared by default.
  3. If you selected the Save filter check box, enter a name for the created filter resource in the Name field. Maximum length of the name: 128 Unicode characters.
  4. In the Conditions settings block, specify the conditions that the events must meet:
    1. Click the Add condition button.
    2. In the Left operand and Right operand drop-down lists, specify the search parameters. Depending on the data source selected in the Right operand field, there may be fields of additional parameters for identifying the value to be passed to the filter. For example, when you select active list, you must specify the name of the active list, the entry key, and the entry key field.
    3. In the operator drop-down list, select an operator.
      Filter operators
      =—the left operand equals the right operand.
      <—the left operand is less than the right operand.
      <=—the left operand is less than or equal to the right operand.
      >—the left operand is greater than the right operand.
      >=—the left operand is greater than or equal to the right operand.
      inSubnet—the left operand (IP address) is in the subnet of the right operand (subnet).
      contains—the left operand contains values of the right operand.
      startsWith—the left operand starts with one of the values of the right operand.
      endsWith—the left operand ends with one of the values of the right operand.
      match—the left operand matches the regular expression of the right operand. The RE2 regular expressions are used.
      hasBit—checks whether the left operand (string or number) contains bits whose positions are listed in the right operand (in a constant or in a list).
      The value to be checked is converted to binary and processed right to left. Chars are checked whose index is specified as a constant or a list.
      
      If the value being checked is a string, then an attempt is made to convert it to integer and process it in the way described above. If the string cannot be converted to a number, the filter returns False.
      
      hasVulnerability—checks whether the left operand contains an asset with the vulnerability and vulnerability severity specified in the right operand.
      If you do not specify the ID and severity of the vulnerability, the filter is triggered if the asset in the event being checked has any vulnerability.
      
      inActiveList—this operator has only one operand. Its values are selected in the Key fields field and are compared with the entries in the active list selected from the Active List drop-down list.
      inDictionary—checks whether the specified dictionary contains an entry defined by the key composed with the concatenated values of the selected event fields.
      inCategory—the asset in the left operand is assigned at least one of the asset categories of the right operand.
      inActiveDirectoryGroup—the Active Directory account in the left operand belongs to one of the Active Directory groups in the right operand.
      TIDetect—this operator is used to find events using CyberTrace Threat Intelligence (TI) data. This operator can be used only on events that have completed enrichment with data from CyberTrace Threat Intelligence. In other words, it can only be used in collectors at the destination selection stage and in correlators.
      inContextTable—presence of the entry in the specified context table.
      intersect—presence in the left operand of the list items specified in the right operand.
    4. If you want the operator to be case-insensitive, select the do not match case check box. The selection of this check box does not apply to the InSubnet, InActiveList, InCategory or InActiveDirectoryGroup operators. This check box is cleared by default.
    5. If you want to add a negative condition, select If not from the If drop-down list.
    You can add multiple conditions or a group of conditions.
  5. If you have added multiple conditions or groups of conditions, choose a selection condition (and, or, not) by clicking the AND button.
  6. If you want to add existing filters that are selected from the Select filter drop-down list, click the Add filter button. You can view the nested filter settings by clicking the button.
After the service is created, you can view and delete spaces in the storage resource settings.

There is no need to create a separate space for audit events. Events of this type (Type=4) are automatically placed in a separate Audit space with a storage term of at least 365 days. This space cannot be edited or deleted from the KUMA Console.
If necessary, in the Disks for cold storage section, add to the storage the disks where you want to transfer events from the ClickHouse cluster for long-term storage.
There can be multiple disks. You can add disks by clicking the Add disk button and remove them by clicking the Delete disk button.

Available settings:
- In the FQDN drop-down list, select the type of domain name of the disk you are connecting:
  - Local—for the disks mounted in the operating system as directories.
  - HDFS—for the disks of the Hadoop Distributed File System.
- In the Name field, specify the disk name. The name must contain 1 to 128 Unicode characters.
- If you select the Local domain name type for the disk, specify the absolute directory path of the mounted local disk in the Path field. The path must begin and end with a "/" character.
- If you select HDFS domain name type for the disk, specify the path to HDFS in the Host field. For example, hdfs://hdfs1:9000/clickhouse/.
Go to the Advanced settings tab and fill in the following fields:
- In the Buffer size field, enter the buffer size in bytes at which events must be sent to the database. The default value is 64 MB. No maximum value is configured. If the virtual machine has less free RAM than the specified Buffer size, KUMA sets the limit to 128 MB.
- In the Buffer flush interval field, enter the time in seconds for which KUMA waits for the buffer to fill up. If the buffer is not full, but the specified time has passed, KUMA sends events to the database. The default value is 1 second.
- In the Disk buffer size limit field, enter a value in bytes. The disk buffer is used to temporarily store events that could not be sent for further processing or storage. If the disk space allocated for the disk buffer is exhausted, events are rotated as follows: new events replace the oldest events written to the buffer. The default value is 10 GB.
- Use the Disk buffer toggle switch to enable or disable the disk buffer. By default, the disk buffer is enabled.
- Use the Write to local database table toggle switch to enable or disable writing to the local database table. Writing is disabled by default.
  If enabled, data is written only on the host on which the storage is located. We recommend using this functionality only if you have configured balancing on the collector and/or correlator — at step 6. Routing, in the Advanced settings section, the URL selection policy field is set to Round robin.
  
  If you disable writing, the data is distributed across the shards of the cluster.
- If necessary, use the Debug toggle switch to enable logging of service operations.
- You can use the Create dump periodically toggle switch at the request of Technical Support to generate resource (CPU, RAM, etc.) utilization reports in the form of dumps.
- In the Dump settings field, you can specify the settings to be used when creating dumps. The specifics of filling in this field must be provided by Technical Support.

The set of resources for the storage is created and is displayed under Resources → Storages. Now you can create a storage service.

Article ID: 221257, Last review: Apr 14, 2025

Page top