Splunk Basic Rules and Search Commands
Let us see the basic search rules and how to work with Splunk search commands.
Launch search app
Now that we’ve included some data into Splunk, we can show you how to search the indexed events. This can be done from the Search app. To launch it, click the Search & Reporting icon:
You should be greeted with the Search summary view:
The Search summary view consists of the following elements:
- App bar – enables you to navigate between the different views in the Search & Reporting app: Search, Pivot, Reports, Alerts, and Dashboards.
- Search bar – used to execute your searches.
- Time range picker – used to select a specific time period that will be searched.
- How to search panel – contains links to the Search Tutorial and Search Manual.
- What to search panel – displays a summary of the data that is installed on this Splunk instance.
- Search history – displays your search history.
To run your searches, you need to enter the search string and hit enter or click the spyglass icon to the right of the time range picker.
Here are the most important rules for searching in Splunk:
- Search terms are case insensitive.
- You can combine multiple search terms in a single search.
- To search for a phrase, use quotation marks. For example, to search for an exact phrase of failed login, you would enter “failed login” in the search bar.
- Boolean logic is supported. You don’t have to write the AND keyword between search terms; it is implied. To specify that either one or two or more arguments should be true, use the OR keyword. To filter out events containing a specific word, use the NOT keyword.
- Splunk’s search language is known as the Search Processing Language (SPL). This language contains hundreds of search commands and their functions, arguments and clauses. For example, to sort results in either ascending or descending order, you would use the SPL command sort. To format results into a tabular output, you can use the table command.
Now that we’ve added data to Splunk and learned the basic rules for searching, we can finally begin to search our events. To search your indexed data, simply type the search term in the Search bar and press enter. Although not required, it is recommended to specify the index you would like to search, as this will ensure a more precise and faster search:
As you can see from the picture above, we’ve searched for the keyword GET in the index called testindex. Splunk found 8,431 events that contain the word GET. Here is the overview of the components in the dashboard:
Timeline – a visual representation of the number of events matching your search over time:
Fields – relevant fields extracted by Splunk. You can use these fields to narrow down your search results:
Results – the events that match your search. Events are ordered by Timestamp, which appears to the left of each event:
Three types of Boolean operators are available in Splunk:
- AND – implied between terms, so you do not need to write it.
- OR – used to specify that either one of two or more arguments should be true.
- NOT – used to filter out events containing a specific word.
NOTE – The Boolean operators listed above must be capitalized (or Splunk will not evaluate them as an operator).
For example, to search for all events that contain either the word GET or the word failed, we would use the following search expression:
To search for all events that contain the word GET but don’t contain the word cart, we would use the following search expression:
It is recommended to use parentheses to group terms. When evaluating Boolean expressions, the precedence is given to terms inside the parentheses.
Keeps (+) or removes (-) fields from search results based on the field list criteria. If + is specified, only the fields that match one of the fields in the list are kept. If - is specified, only the fields that match one of the fields in the list are removed. If neither is specified, defaults to +.
Important: The leading underscore is reserved for names of internal fields such as
_raw and _time. By default, internal fields
_raw and _time are included in the output. The
fields command does not remove internal fields unless explicitly specified with:
... | fields - _*
or more explicitly, with:
... | fields - _raw,_time
Note: Be cautious while removing the
_time field. Statistical commands, such as
chart, cannot display date or time information without the
fields [+|-] <wc-field-list>
- Syntax: <string>, <string>, …
Description: Comma-delimited list of fields to keep (+) or remove (-). You can use wild card characters in the field names.
You can chain consecutive commands together using the pipe character (|). The result of a command to the left of the pipe is used as the input for the next command to the right of the pipe (if you ever worked with the Linux shell, you should be familiar with the concept).
Pipes in Splunk can be used to further filter retrieved events, extract additional event information, evaluate new fields, calculate statistics, sort results, or create a chart. For example, to run a search and display the first 50 results, we can use the following example:
You can stack more than two commands. For example, to run a search, display only first 50 results, and sort results by the host field, we can use the following command:
Time range picker
You can use the time range picker, located to the right of the search bar, to set time boundaries on your searches:
As you can see from the picture above, you can restrict the search to Preset time ranges, Relative time ranges, Real-time ranges or specify a Date Range or a Date & Time Range.
By default, the time range for a search is set to All time. If you know the approximate time range for when the issue has occurred, you can narrow the time range of the search to that time period. For example, if the issue occurred yesterday, you would select Yesterday or Last 24 hours as the time range. You can also define a custom time range. For example, if the issue occurred in the last five hours, you would select the Relative option and specify the earliest time to 5 hours ago:
We’ve already learned that Splunk’s search language is called Search Processing Language (SPL). This language contains hundreds of search commands and their functions, arguments and clauses. In this section, let us see the top command, which can be used to display the most common values of a field, along with their count and percentage.
Consider the following example search:
As you can see on the left, Splunk has extracted fields from the event data. If we want to display the top categories using the top command with the categoryId field:
As you can see from the output above, the top command has displayed the names of the most common categories of the events, along with their count and percentage.
The stats command calculates aggregate statistics over a dataset, such as average, count, and sum. In this section, let us see how to use the stats command to get some useful info about the data.
To display the number of events on each day of the week, we can use the stats count by date_wday command, where date_wday is the name of the field that represents the days in the week:
To display the day of the week with the highest number of events, we can use the stats max(date_wday) command:
To display the day of the week with the lowest number of events, we can use the stats min(date_wday) command:
You can use the sort command to sort the search results by the specified field in either ascending or descending order. For example, to sort the results of our search by the categoryId field, we would use the following command:
Notice how the results were sorted in alphabetical order, with the events with the categoryId of ACCESSORIES coming first.
To sort it in descending order, use a minus sign (-) before the field name:
Notice that now the events containing the categoryId of TEE are displayed first.
where command uses
eval expressions to filter search results. The search keeps only the results for which the evaluation was successful (that is, the Boolean result was true).
where command uses the same expression syntax as
eval. Also, both commands interpret quoted strings as literals. If the string is not quoted, it is treated as a field. Because of this, you can use
where to compare two different fields, which you cannot use
search to do.
- Syntax: <string>
Description: A combination of values, variables, operators, and functions that represent the value of your destination field.
The syntax of the eval expression is checked before running the search, and an exception will be thrown for an invalid expression.
- The result of an eval statement is not allowed to be boolean. If the expression cannot be successfully evaluated for a particular event at search-time, eval erases the value in the result field.
- If the expression references a field name that contains non-alphanumeric characters, it needs to be surrounded by single quotes; for example,
- If the expression references literal strings that contains non-alphanumeric characters, it needs to be surrounded by double quotes; for example,
The where command includes the following functions:
abs, case, ceil, ceiling, cidrmatch, coalesce, commands, exact, exp, floor, if, ifnull, isbool, isint, isnotnull, isnull, isnum, isstr, len, like, ln, log, lower, ltrim, match, max, md5, min, mvappend, mvcount, mvindex, mvfilter, mvjoin, mvrange, mvzip, now, null, nullif, pi, pow, random, relative_time, replace, round, rtrim, searchmatch, sha1, sha256, sha512, sigfig, spath, split, sqrt, strftime, strptime, substr, time, tonumber, tostring, trim, typeof, upper, urldecode, validate..