Filtering and searching events

The Events section of the KUMA web interface does not show any data by default. To view events, you need to define an SQL query in the search field and click the button. The SQL query can be entered manually or it can be generated using a query builder.

Data aggregation and grouping is supported in SQL queries.

You can add filter conditions to an already generated SQL query in the window for viewing statistics, the events table, and the event details area:

• Changing a query from the Statistics window

  To change the filtering settings in the Statistics window:

  1. Open Statistics details area by using one of the following methods:
     • In the drop-down list in the top right corner of the events table select Statistics.
     • In the events table click any value and in the opened context menu select Statistics.

     The Statistics details area appears in the right part of the web interface window.

  2. Open the drop-down list of the relevant parameter and hover your mouse cursor over the necessary value.
  3. Use the plus and minus signs to change the filter settings by doing one of the following:
     • If you want the events selection to include only events with the selected value, click the icon.
     • If you want the events selection to exclude all events with the selected value, click the icon.

  As a result, the filter settings and the events table will be updated, and the new search query will be displayed in the upper part of the screen.

• Changing a query from the events table

  To change the filtering settings in the events table:

  1. In the Events section of the KUMA web interface, click any event parameter value in the events table.
  2. In the opened menu, select one of the following options:
     • If you want the table to show only events with the selected value, select Filter by this value.
     • If you want to exclude all events with the selected value from the table, select Exclude from filter.

  As a result, the filter settings and the events table are updated, and the new search query is displayed in the upper part of the screen.

• Changing a query from the Event details area

  To change the filter settings in the event details area:

  1. In the Events section of the KUMA web interface, click the relevant event.

     The Event details area appears in the right part of the window.

  2. Change the filter settings by using the plus or minus icons next to the relevant settings:
     • If you want the events selection to include only events with the selected value, click the icon.
     • If you want the events selection to exclude all events with the selected value, click the icon.

  As a result, the filter settings and the events table will be updated, and the new search query will be displayed in the upper part of the screen.

After modifying a query, all query parameters, including the added filter conditions, are transferred to the query builder and the search field.

When you switch to the query builder, the parameters of a query entered manually in the search field are not transferred to the builder, so you will need to create your query again. Also, the query created in the builder does not overwrite the query that was entered into the search string until you click the Apply button in the builder window.

In the SQL query input field, you can enable the display of control characters.

You can also filter events by time period. Search results can be automatically updated.

The filter configuration can be saved. Existing filter configurations can be deleted.

Filter functions are available for users regardless of their roles.

For more details on SQL, refer to the ClickHouse documentation. See also KUMA operator usage and supported functions.

Generating an SQL query using a builder

In KUMA, you can use a query builder to generate an SQL query for filtering events.

To generate an SQL query using a builder:

1. In the Events section of the KUMA web interface, click the button.

   The filter constructor window opens.

2. Generate a search query by providing data in the following parameter blocks:

   SELECT—event fields that should be returned. The * value is selected by default, which means that all available event fields must be returned. To make viewing the search results easier, select the necessary fields in the drop-down list. In this case, the data only for the selected fields is displayed in the table. Note that Select * increases the duration of the request execution, but eliminates the need to manually indicate the fields in the request.

   When selecting an event field, you can use the field on the right of the drop-down list to specify an alias for the column of displayed data, and you can use the right-most drop-down list to select the operation to perform on the data: count, max, min, avg, sum.

   If you are using aggregation functions in a query, you cannot customize the events table display, sort events in ascending or descending order, or receive statistics.

   When filtering by alert-related events in drilldown analysis mode, you cannot perform operations on the data of event fields or assign names to the columns of displayed data.

   • FROM—data source. Select the events value.
   • WHERE—conditions for filtering events.

     Conditions and groups of conditions can be added by using the Add condition and Add group buttons. The AND operator value is selected by default in a group of conditions, but the operator can be changed by clicking on this value. Available values: AND, OR, NOT. The structure of conditions and condition groups can be changed by using the icon to drag and drop expressions.

     Adding filter conditions:

     1. In the drop-down list on the left, select the event field that you want to use for filtering.
     2. Select the necessary operator from the middle drop-down list. The available operators depend on the type of value of the selected event field.
     3. Enter the value of the condition. Depending on the selected type of field, you may have to manually enter the value, select it from the drop-down list, or select it on the calendar.

     Filter conditions can be deleted by using the button. Group conditions are deleted using the Delete group button.

   • GROUP BY—event fields or aliases to be used for grouping the returned data.

     If you are using data grouping in a query, you cannot customize the events table display, sort events in ascending or descending order, receive statistics, or perform a retroscan.

     When filtering by alert-related events in drilldown analysis mode, you cannot group the returned data.

   • ORDER BY—columns used as the basis for sorting the returned data. In the drop-down list on the right, you can select the necessary order: DESC—descending, ASC—ascending.
   • LIMIT—number of strings displayed in the table.

     The default value is 250.

     If you are filtering events by user-defined period and the number of strings in the search results exceeds the defined value, you can click the Show next records button to display additional strings in the table. This button is not displayed when filtering events by the standard period.

3. Click the Apply button.

   The current SQL query will be overwritten. The generated SQL query is displayed in the search field.

   If you want to reset the builder settings, click the Default query button.

   If you want to close the builder without overwriting the existing query, click the button.

4. Click the button to display the data in the table.

The table will display the search results based on the generated SQL query.

When switching to another section of the web interface, the query generated in the builder is not preserved. If you return to the Events section from another section, the builder will display the default query.

For more details on SQL, refer to the ClickHouse documentation. See also KUMA operator usage and supported functions.

Manually creating an SQL query

You can use the search string to manually create SQL queries of any complexity for filtering events.

To manually generate an SQL query:

1. Go to the Events section of the KUMA web interface.

   An input form opens.

2. Enter your SQL query into the input field.
3. Click the button.

You will see a table of events that satisfy the criteria of your query. If necessary, you can filter events by period.

Supported functions and operators

• SELECT—event fields that should be returned.

  For SELECT fields, the program supports the following functions and operators:

  • Aggregation functions: count, avg, max, min, sum.
  • Arithmetic operators: +, -, *, /, <, >, =, !=, >=, <=.

    You can combine these functions and operators.

    If you are using aggregation functions in a query, you cannot customize the events table display, sort events in ascending or descending order, or receive statistics.

• FROM—data source.

  When creating a query, you need to specify the events value as the data source.

• WHERE—conditions for filtering events.
  • AND, OR, NOT, =, !=, >, >=, <, <=
  • IN
  • BETWEEN
  • LIKE
  • ILIKE
  • inSubnet
  • match (the re2 syntax of regular expressions is used in queries, special characters must be shielded with "\")
• GROUP BY—event fields or aliases to be used for grouping the returned data.

  If you are using data grouping in a query, you cannot customize the events table display, sort events in ascending or descending order, receive statistics, or perform a retroscan.

• ORDER BY—columns used as the basis for sorting the returned data.

  Possible values:

  • DESC—descending order.
  • ASC—ascending order.
• OFFSET—skip the indicated number of lines before printing the query results output.
• LIMIT—number of strings displayed in the table.

  The default value is 250.

  If you are filtering events by user-defined period and the number of strings in the search results exceeds the defined value, you can click the Show next records button to display additional strings in the table. This button is not displayed when filtering events by the standard period.

  Example queries: SELECT * FROM `events` WHERE Type IN ('Base', 'Audit') ORDER BY Timestamp DESC LIMIT 250 In the events table, all events with the Base and Audit type are sorted by the Timestamp column in descending order. The number of strings that can be displayed in the table is 250. SELECT * FROM `events` WHERE BytesIn BETWEEN 1000 AND 2000 ORDER BY Timestamp ASC LIMIT 250 All events of the events table for which the BytesIn field contains a value of received traffic in the range from 1,000 to 2,000 bytes are sorted by the Timestamp column in ascending order. The number of strings that can be displayed in the table is 250. SELECT * FROM `events` WHERE Message LIKE '%ssh:%' ORDER BY Timestamp DESC LIMIT 250 In the events table, all events whose Message field contains data corresponding to the defined %ssh:% template in lowercase are sorted by the Timestamp column in descending order. The number of strings that can be displayed in the table is 250. SELECT * FROM `events` WHERE inSubnet(DeviceAddress, '00.0.0.0/00') ORDER BY Timestamp DESC LIMIT 250 In the events table, all events for the hosts that are in the 00.0.0.0/00 subnet are sorted by the Timestamp column in descending order. The number of strings that can be displayed in the table is 250. SELECT * FROM `events` WHERE match(Message, 'ssh.*') ORDER BY Timestamp DESC LIMIT 250 In the events table, all events whose Message field contains text corresponding to the ssh.* template are sorted by the Timestamp column in descending order. The number of strings that can be displayed in the table is 250. SELECT max(BytesOut) / 1024 FROM `events` Maximum amount of outbound traffic (KB) for the selected time period. SELECT count(ID) AS "Count", SourcePort AS "Port" FROM `events` GROUP BY SourcePort ORDER BY Port ASC LIMIT 250 Number of events and port number. Events are grouped by port number and sorted by the Port column in ascending order. The number of strings that can be displayed in the table is 250. The ID column in the events table is named Count, and the SourcePort column is named Port.

If you want to use a special character in a query, you need to escape this character by placing a backslash (\) character in front of it.

When creating a normalizer for events, you can choose whether to retain the field values of the raw event. The data is stored in the Extra event field. This field is searched for events by using the LIKE operator.

When switching to the query builder, the query parameters that were manually entered into the search string are not transferred to the builder so you will need to create your query again. Also, the query created in the builder does not overwrite the query that was entered into the search string until you click the Apply button in the builder window.

Aliases must not contain spaces.

For more details on SQL, refer to the ClickHouse documentation. See also the supported ClickHouse functions.

Limited complexity of queries in drilldown analysis mode

During a drilldown analysis, the complexity of SQL queries for event filtering is limited if the Related to alert option is selected from the drop-down list when investigating an alert. If this is the case, only the functions and operators listed below are available for event filtering.

If the All events option is selected from the drop-down list, these limitations are not applied.

• SELECT
  • The * character is used as a wildcard to represent any number of characters.
• WHERE
  • AND, OR, NOT, =, !=, >, >=, <, <=
  • IN
  • BETWEEN
  • LIKE
  • inSubnet

  Examples:

  • WHERE Type IN ('Base', 'Correlated')
  • WHERE BytesIn BETWEEN 1000 AND 2000
  • WHERE Message LIKE '%ssh:%'
  • WHERE inSubnet(DeviceAddress, '10.0.0.1/24')
• ORDER BY

  Sorting can be done by column.

• OFFSET

  Skip the indicated number of lines before printing the query results output.

• LIMIT

  The default value is 250.

  If you are filtering events by user-defined period and the number of strings in the search results exceeds the defined value, you can click the Show next records button to display additional strings in the table. This button is not displayed when filtering events by the standard period.

When filtering by alert-related events in drilldown analysis mode, you cannot group the returned data. When filtering by alert-related events in drilldown analysis mode, you cannot perform operations on the data of event fields or assign names to the columns of displayed data.

Filtering events by period

In KUMA, you can specify the time period to display events from.

To filter events by period:

1. In the Events section of the KUMA web interface, open the Period drop-down list in the upper part of the window.
2. If you want to filter events based on a standard period, select one of the following:
   • 5 minutes
   • 15 minutes
   • 1 hour
   • 24 hours
   • In period

     If you select this option, use the opened calendar to select the start and end dates of the period and click Apply Filter. The date and time format depends on your operating system's settings. You can also manually change the date values if necessary.

3. Click the button.

When the period filter is set, only events registered during the specified time interval will be displayed. The period will be displayed in the upper part of the window.

You can also configure the display of events by using the events histogram that is displayed when you click the button in the upper part of the Events section. Events are displayed if you click the relevant data series or select the relevant time period and click the Show events button.

Saving and selecting events filter configuration

In KUMA, you can save a filter configuration and use it in the future. Other users can also use the saved filters if they have the appropriate access rights. When saving a filter, you are saving the configured settings of all the active filters at the same time, including the time-based filter, query builder, and the events table settings. Search queries are saved on the KUMA Core server and are available to all KUMA users of the selected tenant.

To save the current settings of the filter, query, and period:

1. In the Events section of the KUMA web interface, click the icon next to the filter expression and select Save current filter.
2. In the window that opens, enter the name of the filter configuration in the Name field. The name must contain 128 Unicode characters or less.
3. In the Tenant drop-down list, select the tenant that will own the created filter.
4. Click Save.

The filter configuration is now saved.

To select a previously saved filter configuration:

In the Events section of the KUMA web interface, click the icon next to the filter expression and select the relevant filter.

The selected configuration is active, which means that the search field is displaying the search query, and the upper part of the window is showing the configured settings for the period and frequency of updating the search results. Click the button to submit the search query.

You can click the icon near the filter configuration name to make it a default filter.

Deleting event filter configurations

To delete a previously saved filter configuration:

1. In the Events section of the KUMA web interface, click the icon next to the filter search query and click the icon next to the configuration that you need to delete.
2. Click OK.

The filter configuration is now deleted for all KUMA users.

Supported ClickHouse functions

The following ClickHouse functions are supported in KUMA:

• Arithmetic functions.
• Arrays—all functions except:
  • has
  • range
  • functions in which higher-order functions must be used (lambda expressions (->))
• Comparison functions: all operators except == and less.
• Logical functions: "not" function only.
• Type conversion functions.
• Date/time functions: all except date_add and date_sub.
• String functions.
• String search functions—all functions except:
  • position
  • multiSearchAllPositions, multiSearchAllPositionsUTF8, multiSearchFirstPosition, multiSearchFirstIndex, multiSearchAny
  • like and ilike
• Conditional functions: simple if operator only (ternary if and miltif operators are not supported).
• Mathematical functions.
• Rounding functions.
• Functions for splitting and merging strings and arrays.
• Bit functions.
• Functions for working with UUIDs.
• Functions for working with URLs.
• Functions for working with IP addresses.
• Functions for working with Nullable arguments.
• Functions for working with geographic coordinates.

Search and replace functions in strings, and functions from other sections are not supported.

For more details on SQL, refer to the ClickHouse documentation.