Professional Documents
Culture Documents
Splunk Lab Manual (PDFDrive)
Splunk Lab Manual (PDFDrive)
Manual
Lab #1 Start Splunk Enterprise and Launch Splunk Web
2. Login Screen
Please enter your username and password to be obtained by the instructor
1
2
Lab 2 Navigating Splunk Web
About SplunkHome
Splunk Home is your interactive portal to the data and apps accessible from this
Splunk instance. The main parts of Home include the Splunk Enterprise navigation
bar, the Apps menu, the Explore Splunk Enterprise panel, and a custom default
dashboard (not shown here).
Apps
The Apps panel lists the apps that are installed on your Splunk instance that you
have permission to view. Select the app from the list to open it.
For an out-of-the-box Splunk Enterprise installation, you see one App in the
workspace: Search & Reporting. When you have more than one app, you can drag
and drop the apps within the workspace to rearrange them.
• Click the gear icon to view and manage the apps that are installed in your
Splunk instance.
• Click the plus icon to browse for more apps to install.
The options in the Explore Splunk Enterprise panel help you to get started using
Splunk Enterprise. Click on the icons to open the Add Data view, browse for new apps,
3
open the Splunk Enterprise Documentation, or open Splunk Answers.
Use the Splunk bar to navigate your Splunk instance. It appears on every page in
Splunk Enterprise. You can use it to switch between apps, manage and edit your
Splunk configuration, view system-level messages, and monitor the progress of
search jobs.
The Splunk bar in another view, such as the Search & Reporting app's Search
view, also includes an App menu next to the Splunk logo.
4
Return to Splunk Home
Click the Splunk logo on the navigation bar to return to Splunk Home from any
other view in Splunk Web.
Settings menu
The Settings menu lists the configuration pages for Knowledge objects, Distributed
environment settings, System and licensing, Data, and Authentication settings. If you do
not see some of these options, you do not have the permissions to view or edit them.
User menu
The User menu here is called "Administrator" because that is the default user name
for a new installation. You can change this display name by selecting Edit account
and changing the Full name. You can also edit the time zone settings, select a
default app for this account, and change the account's password. The User menu is
also where you Logout of this Splunk installation.
5
Messages menu
All system-level error messages are listed here. When there is a new message to
review, a notification displays as a count next to the Messages menu. Click the X to
remove the message.
Activity menu
The Activity menu lists shortcuts to the Jobs, Triggered alerts, and System
Activity views.
• Click Jobs to open the search jobs manager window, where you can view and
manage currently running searches.
• Click Triggered Alerts to view scheduled alerts that are triggered. This
tutorial does not discuss saving and scheduling alerts. See "About alerts" in
the Alerting Manual.
• Click System Activity to see Dashboards about user activity and status of the
system.
Help
Click Help to see links to Video Tutorials, Splunk Answers, the Splunk Support
6
Portal, and online Documentation.
Find
Use Find to search for objects within your Splunk Enterprise instance. Find performs
non-case sensitive matches on the ID, labels, and descriptions in saved objects. For
example, if you type in "error", it returns the saved objects that contain the term
"error".
These saved objects include Reports, Dashboards, Alerts, and Data models. The
results appear in the list separated by the categories where they exist.
You can also run a search for error in the Search & Reporting app by clicking
Open error in search.
7
8
Lab 4 Searching the tutorial data
Start searching
In this section, you start searching that tutorial data. This topic discusses
searches that retrieve events from the index.
Before you can start this section, you need to first download and add the tutorial
data.
What to search
Review the tutorial data, which represents a fictitious online game store, called
Buttercup Games. The data summary tells you where the data comes from and
what type of data it is. There are five hosts, eight sources, and three source
9
types. The three source types are Apache web access logs
(access_combined_wcookie), Linux secure formatted logs (secure), and the
vendor sales log (vendor_sales).
Most of this tutorial covers searching the Apache web access logs and
correlating it with the vendor sales logs.
Search assistant
You have data for an online store that sells a variety of games. Try to find out
how many errors have occurred on the site.
1. Open Splunk Search, and type buttercupgames into the search bar.
As you type, the Search Assistant opens. There are two parts to search
assistant: the matching search history and search help.
10
Here, search assistant also provides Steps to help you learn How to Search.
Step 1 explains searches to retrieve events with examples for searching with
terms, quoted phrases, boolean operators, wildcards, and field values. Step 2
introduces how to use search commands.
Search assistant has more uses after you start learning the search language.
When you type in search commands, search assistant displays the command
syntax and usage.
If you do not want search assistant to open automatically, click Auto Open to
toggle it off. You can click the down arrow below the search bar to open it back
up again.
Retrieve events from the index
1. Type in keywords to find errors or failures and use Boolean operators: AND,
OR, NOT.
The asterisk wildcard is used to match terms that start with "fail". These terms
can include: failure, failed, and so on.
11
This search retrieves 427 matching events.
Each time you type keywords and phrases, you implicitly use the search
command to retrieve events from a Splunk index. The search command lets you
use keywords, quoted phrases, field values, boolean expressions, and
comparison expressions to specify which events you want to retrieve.
You can also explicitly invoke the search command later in the pipeline to filter
search results. Read "Use the search command" in the Search manual.
12
Use fields to search
You can not take full advantage of the more advanced search features in Splunk
Enterprise without understanding what fields are and how to use them.
About fields
When you look at the Data Summary in the search view, you see tabs for the
Hosts, Sources, and Source Types that described the type of data you added to
your Splunk index.
These are also default fields (host, source, sourcetype) that Splunk Enterprise
extracts from the data during indexing. They help to specify exactly which events
you want to retrieve from the index.
Fields exist in machine data in many forms. Often, a field is a value (with a fixed,
delimited position on the line) or a name and value pair, where there is a single
value to each field name. A field can be multivalued, that is, it can appear more
than once in an event and has a different value for each appearance.
Some examples of fields are clientip for IP addresses accessing your Web
server, _time for the timestamp of an event, and host for domain name of a
server. One of the more common examples of multivalue fields is email address
13
fields. While the From field will contain only a single email address, the To and Cc
fields have one or more email addresses associated with them.
In Splunk Enterprise, fields are searchable name and value pairings that
distinguish one event from another because not all events will have the same
fields and field values. Fields let you write more tailored searches to retrieve the
specific events that you want.
Extracted fields
Default and other indexed fields are extracted for each event that is processed
when that data is indexed. Default fields include host, source, and sourcetype..
Splunk Enterprise extracts different sets of fields, when you run a search..
You can also use the field extractor to create custom fields dynamically on your
local Splunk instance. The field extractor lets you define any pattern for
recognizing one or more fields in your events.
1. Go to the Search dashboard and type the following into the search bar:
sourcetype="access_*"
Search for fields use the syntax: fieldname="fieldvalue" . Field names are
case sensitive, but field values are not. You can use wildcards in field values.
Quotes are required when the field values include spaces.
This search indicates that you want to retrieve only events from your web access
logs and nothing else.
14
This search uses the wildcard access_* to match any Apache web
access sourcetype, which can be access_common, access_combined, or
access_combined_wcookie.
If you are familiar with the access_combined format of Apache logs, you
recognize some of the information in each event, such as:
15
These are events for the Buttercup Games online store, so you might recognize
other information and keywords, such as Arcade, Simulation, productId,
categoryId, purchase, addtocart, and so on.
To the left of the events list is the Fields sidebar. As Splunk Enterprise retrieves
the events that match your search, the Fields sidebar updates with Selected
fields and Interesting fields. These are the fields that Splunk Enterprise
extracted from your data.
16
Selected Fields are the fields that appear in your search results. The default
fields host, source, and sourcetype are selected. These fields appear in all the
events.
You can hide and show the fields sidebar by clicking Hide Fields and Show
Fields, respectively.
The Select Fields dialog box opens, where you can select the fields to show in
the events list.
17
You see more default fields, which includes fields based on each event's
timestamp (everything beginning with date_*), punctuation (punct), and
location (index).
Other field names apply to the web access logs. For example, there are
clientip, method, and status. These are not default fields. They are
extracted at search time.
Other extracted fields are related to the Buttercup Games online store. For
example, there are action, categoryId, and productId.
4. Select action, categoryId, and productId and close the Select Fields
window.
The three fields appear under Selected Fields in the sidebar. The selected fields
appear under the events in your search results if they exist in that particular
event. Every event might not have the same fields.
18
The fields sidebar displays the number of values that exist for each field. These
are the values that Splunk Enterprise indentifies from the results of your search.
In this set of search results, Splunk Enterprise found five values for action, and
that the action field appears in 49.9% of your search results.
6. Close this window and look at the other two fields you selected, categoryId
(what types of products the shop sells) and productId (specific catalog number
for products).
If you click on the arrow next to an event, it opens up the list of all fields in that
event.
19
Use this panel to view all the fields in a particular event and select or deselect
individual fields for an individual event.
Example1: Search for successful purchases from the Buttercup Games store.
20
sourcetype=access_* status=200 action=purchase
This search uses the HTTP status field, status, to specify successful requests
and the action field to search only for purchase events.
Example 3: Search for how many simulation games were bought yesterday.
Select the Preset time range, Yesterday, from the time range picker and run:
categoryId=simulation
The count of events returned are the number of simulation games purchased.
To find the number purchases for each type of product sold at the shop, run this
search for each unique categoryId. For the number of purchases made each day
of the previous week, run the search again for each time range.
21
Use the search language
The searches you have run to this point have retrieved events from your Splunk
index. You were limited to asking questions that could only be answered by the
number of events returned.
For example, in the last topic, you ran this search to see how many simulation
games were purchased:
To find this number for the days of the previous week, you have to run it against
the data for each day of that week. To see which products are more popular than
the other, you have to run the search for each of the eight categoryId values and
compare the results.
This section explains in more detail one of the ways you can use the search
assistant to learn about the Splunk search processing language and construct
searches.
22
You've seen before that search assistant displays type ahead for keywords
that you type into the search bar. It also explains briefly how to search.
The pipe indicates to Splunk that you're about to use a command, and that you
want to use the results of the search to the left of the pipe as the input to this
command. You can pass the results of one command into another command in a
series, or pipeline, of search commands.
23
You want Splunk to give you the most popular items bought at the online store.
24
4. Type the categoryId field into the search bar to complete your search.
The count of events under the search bar indicates the number of events
retrieved that match the search for sourcetype=access_* status=200
action=purchase. The results of the top command appear in the Statistics
tab.
25
View reports in the Statistics tab
The top command also returns two new fields: count is the number of times each
value of the field occurs, and percent is how large that count is compared to the
total count.
You can also view the results of transforming searches in the Visualizations tab
where you can format the chart type. For example, a search using the top
command can be illustrated with a pie chart.
26
By default, the Visualizations tab opens with a Column Chart.
Column, Bar, and Pie charts are recommended for this data set.
3. Select Pie.
27
You can turn on drill down to delve deeper into the details of the information
presented to you in the tables and charts that result from your search.
4. Mouse over each slice of the pie to see the count and percentage values for
each categoryId.
28
5. Click on a slice, such as "Strategy".
29
30
31
Lab 5 Subsearchs
Use a subsearch
This topic walks you through examples of correlating events with subsearches.
Let's try to find the single most frequent shopper on the Buttercup Games online
store and what this customer has purchased.
To do this, search for the customer who accessed the online shop the most.
Limit the top command to return only one result for the clientip.
32
This search returns one clientip value, which we'll use to identify our
VIP customer.
This search used the count() function which only returns the total count of
purchases for the customer. The dc() function is used to count how many
different products he buys.
The drawback to this approach is that you have to run two searches each time
you want to build this table. The top purchaser is not likely to be the same person
at any given time range.
clientip
Here, the subsearch is the segment that is enclosed in square brackets, [ ]. This
search, search sourcetype=access_* status=200 action=purchase | top
33
limit=1 clientip | table clientip is the same as Example 1 Step 1, except
for the last piped command, | table clientip
Because the top command returns count and percent fields as well, the table
command is used to keep only the clientip value.
34
These results should match the previous result, if you run it on the same time range.
But, if you change the time range, you might see different results because the top
purchasing customer will be different.
35
Lab 3 Add the sample data into Splunk Enterprise
If you are not in Splunk Home, click the Splunk logo on the
Splunk bar to go to Splunk Home.
The Add Data view opens. The Add Data displays three options for
adding data, lists of common data types, and add-ons you can use
to extend Splunk Enterprise's capabilities to add data.
36
2. Under Select Source, click Select File
to browse for the tutorial
data or Drop the data file into the outlined box.
Because the tutorial data file is an archived data file, the next
step in the Add Data workflow changes from Set Sourcetype
to Input Settings.
Under Input Settings, you can override the default settings for
Host, Source type, and Index.
37
4. Modify the host settings to assign host names using a portion of
the path name:
38
6. Click Submit
7. .( PLEASE
DO NOT SUBMIT BECAUSE INSTRUCTOR ALREADY
SUBMITTED FOR ENTIRE CLASS)
39
40
Lab 6 Field Lookup
This topic takes you through using field lookups to add new fields to your events.
Field lookups let you reference fields in an external CSV file that match fields in
your event data. Using this match, you can enrich your event data by adding
more meaningful information and searchable fields to each event.
41
This opens the Lookups editor where you can create new lookups or edit existing
ones.
1. In the Lookups manager under "Actions" for Lookup table files, click Add
new.
42
This takes you to the Add new' lookup table files view where you upload CSV
files to use in your definitions for field lookups.
2. To save your lookup table file in the Search app, leave the Destination app as
search.
3. Under Upload a lookup file, browse for the CSV file (prices.csv) to upload.
This is the name you use to refer to the file in a lookup definition.
5. Click Save.
This uploads your lookup file to the Search app and returns to the lookup table
files list.
43
Note: If Splunk does not recognize or cannot upload the file, check that it was
uncompressed before you attempt to upload it again.
If the lookup file is not shared, you can not select it when you define the lookup.
44
2. Under Sharing for the prices.csv lookup table's Path, click Permissions.
This opens the Permission dialog box for the prices.csv lookup file.
4. Click Save.
This takes you to the Add new lookups definitions view where you define your
field lookup.
45
3. Leave the Destination app as search.
46
File-based lookups add fields from a static table, usually a CSV file.
6. Under Lookup file, select prices.csv (the name of your lookup table).
8. Click Save.
47
4. Click Save.
1. In the Lookups manager, under Actions for Automatic lookups, click Add
New.
48
This takes you to the Add New automatic lookups view where you configure the
lookup to run automatically.
49
5. Under Apply to and named, select sourcetype and type in
access_combined_wcookie.
6. Under Lookup input fields type in productId in both text areas under
Lookup input fields .
Splunk Enterprise matches the field in the lookup table (which is the one
specified on the left) with the field on the right (which is the field in your events).
50
In this case the field names match.
7. Under Lookup output fields, type in the name of the fields that you want to
add to your event data based on the input field matching and rename the fields.
7.1 In the first text area, type product_name, which contains the descriptive name
for each productId.
7.2. In the second text area, after the equal sign, type productName. This
renames the field to productName.
7.3. Click Add another field to add more fields after the first one.
7.4. Add the field price, which contains the price for each productId. Do not
rename this field.
9. Click Save.
This returns you to the list of automatic lookups and you should see your
configured lookup.
51
Show the new fields in your search results
1. Return to Search.
3. Scroll through the list of Interesting Fields in the fields sidebar, and find the
price field.
52
5. Next to Selected, click Yes.
The price field appears under Selected Fields in the fields sidebar.
1. Copy and paste or type in the previous subsearch example to see what the
VIP customer bought. This time, replace the productId field with productName.
53
table clientip] | stats count AS "Total Purchased", dc(productId) AS
"Total Products", values(productName) AS "Product Names" by clientip |
rename clientip AS "VIP Customer"
The result is the same as in the previous subsearch example, except that the VIP
customer's purchases are more meaningful with the added descriptive product names.
54
Lab 7 Saving and sharing Reports
This lab takes you through saving searches and more search examples.
Save as a report
1. Select the time range Yesterday and run the following search
2.
sourcetype=access_* status=200 action=purchase [search
sourcetype=access_* status=200 action=purchase | top limit=1 clientip |
table clientip] | stats count AS "Total Purchased", dc(productId) AS
"Total Products", values(productName) AS "Product Names" by clientip |
55
5. (Optional) Enter a Description Buttercup Games most frequent shopper.
8. Click Save.
56
There are other options in this window.
• Continue Editing lets you refine the search and report format.
• Add to dashboard lets you add the report to a new or existing dashboard.
• View lets you view the report.
9. Click View.
You can view and edit the saved report from its report view.
57
You can open the report in the search view and edit the saved search's
description, permissions, schedule, and acceleration. You can also clone,
embed, and delete the report from this menu.
You can view and edit different properties of the report, including its schedule,
acceleration, permissions, and embedding.
You saved this report with a time range picker. The time range picker lets you
change the time period to run this search. For example, you can use this time
range picker to run this search for the VIP Customer Week to date, Last 60
minutes, Last 24 hours just by selecting the Preset time range or defining a
custom time range.
58
Find and share saved reports
You can access your saved reports using the app navigation bar.
When you save a new report, its Permissions are set to Private. This means
that only you can view and edit the report. You can allow other apps to view, or
59
edit, or view and edit the reports by changing its Permissions.
1. Under Actions for the VIP Customer report, click Edit and select Edit
Permissions.
2. In the Edit Permissions dialog box, set Display For to App and check the
box under Read for Everyone.
60
This action gives everyone who has access to this app the permission to view it.
3. Click Save.
Back at the Reports listing page, you see that the Sharing for VIP Customer now
reads App.
61
About report acceleration
If your search has a large number of events and is slow to finish, you might be able
to accelerate the resulting report so it finishes faster when you run it again. This
option is available when the report produced by your search qualifies for
acceleration. The "VIP Customer" report does not qualify for acceleration, because it
is based on a transforming search.
62
Lab 8 More Searches and Reports
In this example, calculate the number of views, purchases, and adds to cart for
each type of product.
This report requires the productName field from the fields lookup example. If
you did not add the lookup, refer to that example and follow the procedure.
63
This search uses the chart command to count the number of events that are
action=purchase and action=addtocart.
2. Use the Visualization view options to format the results as a column chart.
purchases AS "Purchases"
Instead of the chart command, this search uses the stats command to count the
user actions. Then, it uses the eval command to define two new fields which
64
calculate conversion rates for "Product Views to Purchases" and "Adds to cart to
Purchases".
Steps 2 to 6 reformat the visualization to overlay the Conversion series onto the
Actions series.
2. Click Visualization.
65
3. Click Format and X-Axis.
4.
4.1 Rotate the label -45 degrees and do not truncate the label.
66
5. Click Format and Y-Axis.
5.2 Set the Max Value to 2500 and the Interval to 500.
67
6.1 Type in or select the fields, "viewsToPurchase" and "cartToPurchase".
6.5 Set the Max Value to 100 and the Interval to 20.
68
7. Click Save As and select Report.
7.1 In the Save Report As dialog box, enter a Title, "Comparison of Actions and
Conversion Rates by Product".
8. Click Save.
69
Example 3: Products purchased over time
For this report, chart the number of purchases that were completed for each item.
This report requires the productName field from the fields lookup example. If you
didn't add the lookup, refer to that example and follow the procedure.
1. Search for:
Use the count() function to count the number of events that have the field
action=purchase. Use the usenull and useother arguments to make sure the
chart counts events that have a value for productName.
70
2. Click the Visualization tab and Format the X-Axis, Y-Axis, and Legend to
produce the following line chart.
71
3.1 In the Save Report As dialog box, enter a Title, "Product Purchases over
Time".
3.2 (Optional) Enter a Description, "The number of purchases for each product."
72
Example 4: Purchasing trends
This example uses sparklines to trend the count of purchases made over time.
For stats and chart searches, you can add sparklines to their results tables.
Sparklines are inline charts that appear within the search results table and are
designed to display time-based trends associated with the primary key of each row.
See "Add sparklines to your search results" in the Search Manual.
This example requires the productName field from the fields lookup example. If
you did not add the lookup, refer to that example and follow the procedure.
This search uses the chart command to count the number of purchases,
action="purchase", made for each product, productName. The difference is that the
count of purchases is now an argument of the sparkline() function.
73
4. In the Save Report As dialog box, enter a Title, "Purchasing trends".
74
Lab 9 Dashboards
About dashboards
Dashboards are views that are made up of panels that can contain modules
such as search boxes, fields, charts, tables, and lists. Dashboard panels are
usually hooked up to saved searches.
After you create a visualization or report, you can add it to a new or existing
dashboard using the Save as report dialog box. You can also use the
Dashboard Editor to create dashboards and edit existing dashboards. Using the
Dashboard editor is useful when you have a set of saved reports that you want to
quickly add to a dashboard.
You can specify access to a dashboard from the Dashboard Editor. However,
your user role (and capabilities defined for that role) might limit the type of access
you can define.
If your Splunk user role is admin (with the default set of capabilities), then you
can create dashboards that are private, visible in a specific app, or visible in all
apps. You can also provide access to other Splunk user roles, such as user,
admin, and other roles with specific capabilities.
After you create a panel with the Dashboard Editor, use the Visualization Editor
to change the visualization type in the panel, and to determine how that
visualization displays and behaves. The Visualization Editor lets you choose from
visualization types that have their data structure requirements matched by the
search that has been specified for the panel.
75
Edit the XML configuration of a dashboard
Although you are not required to use XML to build dashboards, you can edit a
dashboard's panels by editing the XML configuration for the dashboard. This
provides editing access to features not available from the Dashboard Editor. For
example, edit the XML configuration to change the name of dashboard or specify
a custom number of rows in a table.
This topic walks you through saving a search as a dashboard panel and adding
an input element to the dashboard.
76
2. Click the Visualization tab and select the Pie chart type.
77
4.1. For Dashboard, click New.
4.2. Enter the Dashboard Title, "Buttercup Games Purchases", The Dashboard
ID updates with "Buttercup_games_purchases".
5. Click Save.
78
6. Click View Dashboard.
This creates a dashboard with one report panel. To add more report panels, you
can run new searches and save them to this dashboard, or you can add saved
reports.
79
You can Create a new dashboard and edit existing dashboards. You see the
Buttercup Games Purchases dashboard that you created.
2. Under the i column, click the arrow next to Buttercup Games Purchases to
see more information about the dashboard: What app context it is in, whether or
not it is scheduled, and its permissions.
You can use the quick links that are inline with the information to edit the
dashboard's Schedule and Permissions.
80
2. Click Edit and select Edit Panels.
In this view, you have edit buttons: Add Input, Add Panel, and Edit Source.
81
This adds a time range picker input to the dashboard editor.
4. Click the Edit Input icon for the time range picker. It looks like a pencil.
This opens a set of input controls. The Time input type should be preselected.
This optional step redefines the name of the input token for the time range picker.
Because the default names of input tokens are not very descriptive (field1, field2,
82
field3, and so on), you may want to do this when you give your dashboard
multiple inputs. It makes it easier to understand which input you are working with.
You can also optionally change the default time range for the picker by changing
the value of Default. Right now it defaults to All time.
In the next two steps you connect your dashboard panel to this time range picker.
6. In the new dashboard panel, click the Inline Search icon and select
Edit Search String.
8. Click Save.
83
The panel is now hooked up to the shared time range picker input. The inline
search that powers the panel now uses the time range selected for the shared
time range picker.
As you add panels to this dashboard, repeat steps 6 through 8 to hook the new
panels up to the shared time range picker input.
You can have dashboards that offer a mix of panels that work with the shared
time range picker and panels that show data for fixed time ranges.
In the previous section, you ran searches and saved them as reports. In this
topic, you add the saved reports to an existing dashboard.
84
2. Click Edit and select Edit Panels.
85
3. In the Buttercup Games Purchases dashboard editor, click Add Panel.
4.
86
6. Select Purchasing Trends.
87
The new panel is placed in the dashboard editor. You can click anywhere to
close the Add Panel sidebar menu or choose another report to add to the
dashboard.
Note: If you want the new panels to work with the shared time range picker input,
repeat steps 6 through 8 from the "Add an input to the dashboard" procedure to
connect them to that input.
88
9. Close the Add Panel sidebar and rearrange the panels on the dashboard.
While in the dashboard editor, you can drag and drop a panel to rearrange it on
the dashboard.
89
More dashboard actions
After you complete the dashboard, you can Export to PDF and Print the
dashboards using the buttons to the upper right. You can also share the
dashboard with other users by changing its permissions.
90
Lab 10 Create a new data model
This topic shows you how to create new data models based on the tutorial data. Data
models are created within Pivot and you need to have admin or power role to create
a data model.
By default only users with the admin or power role can create data models. For
other users, the ability to create a data model is tied to whether their roles have
"write" access to an app. Since this is a first time install, you have admin privileges
by default and should be able to continue.
If you are not able to create or edit a data model, you may need to check your
permissions. For more information, read "About data model permissions" in the
Knowledge Manager Manual.
91
2. Under Knowledge, click Data Models.
This takes you to the Data Models management page. The Data Models
management page is a listing page of data models. If you have existing data models
in this Splunk Enterprise instance, this page lists them. Use this page to manage the
permissions, acceleration, cloning, and removal of existing data models. You can
also use this page to upload a data model or create new data models, using the
Upload Data Model and New Data Model buttons on the top right.
92
2. Enter the Title, "Buttercup Games"
The Title field accepts any character, as well as spaces. The value you enter here
is what appears on the data model listing pages.
The ID must be a unique identifier for the data model. It cannot contain spaces or any
characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z, 0-9,
_, or -). Spaces between characters are also not allowed. Once you define the data
model ID, you can't change it.
5. (Optional) Enter the Description, "Enables data analysis and reporting for
tutorial data."
6. Click Create.
Use this page to create objects for the new data model, define their constraints and
attributes, arrange the objects in logical hierarchies, and manage them.
93
Lab 11 Define a root object for the data model
In the last lab, you created the data model "Buttercup Games".
This lab walks you through adding a root object for Buttercup Games
purchases.
Use the Edit Objects page to design a new data model or redesign an existing
data model. On the Edit Objects page, you can create objects for your data
model, define their constraints and attributes, arrange them in logical object
hierarchies, and maintain them.
Data models are typically composed of object hierarchies built on root event
objects. Each root event object represents a set of data that is defined by a
94
constraint, which is a simple search that filters out events that are not relevant to
the object.
1. To define the data model's first event base object, click Add Object.
Your first root object can be either a Root event or Root search.
95
3. Enter the Object Name: Purchase Requests
The Object Name field can accept any character, as well as spaces. It's what
you'll see on the Choose an Object page and other places where data model
objects are listed.
This should automatically populate when you type in the Object Name. You can
edit it if you want to change it.
The Object ID must be a unique identifier for the object. It cannot contain spaces
or any characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z,
0-9, _, or -). Spaces between characters are also not allowed. Once you save the
Object ID value, you can't edit it.
This defines the web access page requests that are purchase events.
After you provide Constraints for the event base object you can click Preview to
test whether the constraints you've supplied return the kinds of events you want.
96
6. Click Save.
97
The list of attributes for the root object include: host, source, sourcetype, and
_time. If you want to add child objects to client and server errors, you need to edit the
attributes list to include additional attributes.
98
Lab 12 Designing a Pivot report
About Pivot
The Splunk Enterprise Pivot tool lets you quickly design reports with tables and
data visualizations that present different aspects of a selected Data Model. Pivot
lets you generate these reports with a UI interface instead of having to use the
search processing language.
Pivot views
Entering Pivot takes you to the Select a Data Model page, where you should
see a list of the data models if any have been created. For example, this list
99
includes the Buttercup Games data model that you created earlier in this
tutorial. It also includes two sample data models that track Splunk Enterprise
internal and audit logs.
If you view Pivot in smaller browser windows, the Search & Reporting app's
navigation bar is hidden. To use the navigation bar, click the menu icon on the
upper right. The navigation bar slides down.
100
2. Use the arrows under the i column to view information for Buttercup Games.
Clicking Edit objects takes you to the object editor for the Buttercup Games data
model.
This takes you to the Select a Data Object view. This view lists all the objects
that have been created for this data model. The Buttercup Games data model
consists of the Purchase Requests parent object and the Successful Purchases
and Failed Purchases child objects.
4. Use the arrows under the i column to view the information for the objects.
101
5. Click Purchase Requests.
102
Selecting an object from the Select a Data Object view takes you to the New
Pivot editor for that data model.
Components of Pivot
Visualization types: The left-hand vertical bar contains icons that represent
different visualization types. Selecting a different icon controls which Pivot builder
and report interfaces display. Visualization types are: Statistics Table (default),
Column Chart, Bar Chart, Scatter Chart, Bubble Chart, Area Chart, Line Chart,
Pie Chart, Single Value Display, Radial Gauge, Marker Gauge, and Filler Gauge.
103
dashboard panel (Dashboard Panel).
• Clear: Reset the interface to its initial state, which will dismiss the saved
report (if applicable), change the visualization type to Statistics Table, and
populate the report with a single Column Value for the count of the object
and a time filter for all time (if _time is an applicable field).
• Data model object: This is the right-most button. It takes its label from the
data model object that was selected. For example, in the screenshot it is
"Purchase Requests". Use this menu to navigate back to the list of data
models (Select another Data Model), navigate back to the list of data
model objects (Select another Object), or edit the selected data model
object (Edit Object). Additionally, you can rebuild acceleration and inspect
the acceleration job.
104
Job Actions: The Pause and Stop buttons control the progress of the Pivot job.
Other actions include: Share, Export, Print, and Open in Search. Clicking
Open in Search opens the Search view and runs the current search string.
105
Create and save a Pivot
This topic shows you how to use pivot to create and save a simple report.
This is a very simple example. More complicated examples are shown in later
topics of this tutorial.
When you set out to design a report, you first need to select a data model that
represents the broad category of event data that you want to work with. For this
tutorial, that data model is the "Buttercup Games".
106
1. From the app navigation bar, select Pivot to enter the Select a Data Model
page.
The Buttercup Games data model has a root object to track Purchase Requests
from the game website. The Purchases object breaks down into Successful and
Failed purchases.
This opens a New Pivot editor for the Purchase Requests object.
107
By default, the Pivot Editor interface displays elements to define a pivot table.
There are four basic pivot element categories: Filters, Split Rows, Split Columns,
and Column Values. When you first open the Pivot Editor for a specific object,
only two elements will be defined:
This gives you the single value, which is the total count of events returned by the
object over all time. In this case, this count is the "Count of Purchase Requests".
4. Select the Single Value Display element from the visualization bar.
108
4.a Next to Under Label, type Purchase Requests.
After you define a pivot, you can save it as either a report or a dashboard panel.
In this example, you save the single value display as a report. Dashboards and
dashboard panels are discussed in a later chapter.
109
The Save as Report dialog box opens.
110
3. Select Yes to include the time range picker. (This should be the default.)
4. Click Save.
After the report saves, a window displays that "Your report has been created". You
can continue editing the current Pivot, add the pivot to a dashboard, change
additional settings for the saved report, or view the report.
111
View saved reports
A report that is created from Pivot will always be saved under the current app and
owner namespace.
1. Click Reports in the app navigation bar to view the list of all saved reports.
2. Use the arrow in the i column to view information about Total Purchase
Requests report.
112
Lab 13 Pivots and Visualizations
In the previous topic you used pivot to find the total number of purchase requests
and saved the single value display as a report. In this topic, you will use the pivot
visualization editor to create a pivot table of the Buttercup Games Successful
Purchases object.
The Successful Purchases object has attributes for the products purchased from
the Buttercup Games website. This includes the automatically extracted
attributes (categoryId and productId) as well as the lookup attributes (price and
product_name).
The Buttercup Games online store offers hundreds of products, of a variety of
categories, and you want to know more about the items that were purchased
over the past week. You can create a pivot report that breaks down the total
number of purchase events by product name, and through that quickly see which
of your products were the top sellers for that period.
1. From the app navigation bar, select Pivot to enter the "Select a Data Model"
page.
2. Choose the Buttercup Games data model and select the Successful
Purchases child object.
113
The New Pivot editor for Successful Purchases opens.
You can add multiple elements from each pivot element category to define your
pivot table. It's easy to add, define, and remove pivot elements in the process of
determining what information your table should provide.
• To add a pivot element: Click the + icon. This opens up the element
dialog, where you choose an attribute and then define how the element
114
uses that attribute.
• To inspect or edit an element: Click the "pencil" icon on the element.
This opens the element dialog.
• To reorder and transfer pivot elements: Drag and drop an element
within its pivot element category to reorder it. Drag and drop elements
between element categories to transfer them.
• To remove pivot elements from the Pivot Editor: Open its element
dialog and click the Remove button, or drag the element up or down until
it turns red and drop it.
Under Filters, the time filter is always present when you build a pivot; you cannot
remove it. It defines the time range for which the pivot returns results. It operates
exactly like the time range menu that is in use throughout Splunk Web.
Currently your Pivot table shows a single value, the total count of Successful
Purchases over All time.
Change the time filter to view the Successful Purchases over a different time
range:
1. Under Filter, click the pencil next to All time to open the time range picker.
115
2. Under Presets and Relative, click "Last 7 days".
(If this shows no events, you can select "All time" and continue.)
Add Pivot elements to see the Count of Successful Purchases for each product
by name:
1. Under Split Rows, click + and select productName, the lookup field that
contains the name of each product, based on the productId.
116
This opens a dialog box that lets you format the field.
117
Add a Column Value element
Add a Column Value to see total earned for each product that was successfully
purchased:
118
2.a Enter the label Total Revenue.
2.b Select the Value Sum.
This creates a field called Total Revenue, which is the summation of the price
for each successful purchase of the product. (You can add the price values as
another Split Row, if you want to see the cost of each individual product in this
table.)
119
Save the Pivottable
3. Click Save.
4. In the Your Report Has Been Created dialog box, click View.
120
Create a pivot chart
In the previous topic you used Pivot visualization editor to build a table. In this
topic, you will use the same object to create chart visualizations.
1. From the app navigation bar, select Pivot to enter the "Select a Data Model"
page.
2. Choose the Buttercup Games data model and select the Successful
Purchases child object.
121
The New Pivot editor for Successful Purchases opens.
Visualization types are listed in the black sidebar that runs down the left-hand
side of the Pivot editor. By default, the statistics table visualization is selected
when you enter Pivot.
It can be helpful to begin building your pivot as a table and then switch over to
the visualization of your choice. When you switch between pivot visualization
types, Pivot will find the elements it needs to create the visualization, discard the
elements it does not need, and notify you when needed elements need to be
defined. This happens when you switch between tables and charts as well as
between chart types.
In the last topic, we looked at purchases by product ID and name. Now, let's
report on the count of successful purchases by category.
122
Add a Split Row for the categoryId field.
1. Under Split Rows, click + and select categoryId from the list.
123
Change the visualization type
124
• Column charts use the first split row element in pivot table definitions to
provide their X-axis values. In this case, that Split Row is Category.
• Column charts use the first column value element in pivot table definitions to
provide their Y-axis values. Here, that Column Value is Count of Successful
Purchases.
125
The New Pivot editor for the Column chart displays.
• Pie charts use the values from the first Split Row element (Category) to
determine the number and colors of their slices.
• Pie charts use the first Column Value element (Count of Successful
Purchases) to determine the relative sizes of their slices.
126
Mouseover a slice of the pie chart to view the metrics: Category, Count of
Successful Purchases, and percentage of the total Count of Successful
Purchases.
127
Lab 14 Pivots and Dashboards
Creating a dashboard
About dashboards
You can specify access to a dashboard from the Dashboard Editor. However,
your user role (and capabilities defined for that role) may limit the type of access
you can define.
If your user role is admin (with the default set of capabilities), then you can create
dashboards that are private, visible in a specific app, or visible in all apps. You
can also provide access to other user roles, such as user, admin, and other roles
with specific capabilities.
128
Change dashboard panel visualizations
After you create a panel with the Dashboard Editor, use the Visualization Editor
to change the visualization type displayed in the panel, and to determine how
that visualization displays and behaves. The Visualization Editor only allows you
to choose from visualization types that have their data structure requirements.
You just created a pie chart in the last lab, now let's save it to a dashboard
panel.
129
2. Define a new dashboard to save the panel to:
4. Click Save.
130
View and edit dashboard panels
After you save a dashboard, you can access it by clicking Dashboards in the
app navigation bar.
You can Create a new dashboard and edit existing dashboards. You see the
Buttercup Games dashboard you just created.
2. Under the i column, click the arrow next to Buttercup Games to see more
information about the dashboard: What app context it is in, whether or not it is
scheduled, and its permissions.
131
There are also quick links to edit the dashboard's Schedule and Permissions
inline with the information.
To view the dashboard, click the dashboard's Title or select the Edit option
under Actions.
Note: If you click to view a dashboard and you cannot view it (or it displays
blank), check that you have read access to the data model. To do this, go to the
Manage Data Models view and edit the Permissions for the Buttercup Games
data model to share in the App.
132
In this view, you have edit buttons: Add Input, Add Panel, and Edit Source.
This adds a shared time range picker input to the dashboard editor.
133
4. Click the Edit Input icon for the time range picker. It looks like a pencil.
This opens a set of input controls. The Time input type should be preselected.
This optional step redefines the name of the input token for the time range picker.
Because the default names of input tokens are not very descriptive (field1, field2,
134
field3, and so on), you may want to do this when you give your dashboard
multiple inputs. It makes it easier to understand which input you are working with.
You can also optionally change the default time range for the picker by changing
the value of Default. Right now it defaults to All time.
In the next two steps you connect your dashboard panel to this time range picker.
6. In the new dashboard panel, click the Inline Pivot icon and select Edit
Search String.
8. Click Save.
135
The panel is now hooked up to the shared time range picker input. The inline
search that powers the panel now uses the time range selected for the shared
time range picker.
As you add panels to this dashboard, repeat steps 6 through 8 to hook the new
panels up to the shared time range picker input.
You can have dashboards that offer a mix of panels that work with the shared
time range picker and panels that show data for fixed time ranges.
Add another panel using one of the saved reports you created earlier.
1. In the Buttercup Games dashboard, click Edit and select Edit Panels.
136
2. In the Edit: Buttercup Games view, click Add Panel.
137
4. Click Total Purchase Requests.
This slides open a preview panel with information about the saved report.
The new panel is placed in the dashboard editor. You can click anywhere to
close the Add Panel sidebar menu or choose another report to add to the
dashboard.
138
Before you close the Add Panel sidebar menu, add a second report.
While in the dashboard editor view, drag and drop the panels to rearrange them
on the dashboard.
7. Click Done.
139
140
Lab 15 Alerts
Create alerts
A scheduled alert evaluates the results of a historical search that runs over a
specified time range on a regular schedule. The alert fires when it encounters the
trigger condition.
For example, you can create a scheduled alert to monitor online sales. The
search runs daily at midnight and triggers when the sum of the sales of a specific
item is below 500 for the previous day. When the alert triggers, it sends an email
to the appropriate administrators monitoring sales.
1. From the Search Page, create the following search. Select Last 24 Hours
for the time range:
141
♦ Add Actions: List in Triggered Alerts
See Set up alert actions for information on other actions.
6. Click Save.
When scheduling an alert, you can use cron notation for customized schedules.
When specifying a cron schedule, only five cron parameters are available, not
six. The sixth parameter for year, common in other forms of cron notation, is not
available.
* * * * *
correspond to:
When you select Run on Cron Schedule for the time range of a scheduled alert,
enter the earliest and latest parameters for a search. What you enter overrides
the time range you set when you first ran the search.
To avoid overlaps or gaps, the execution schedule should match the search time
range. For example, to run a search every 20 minutes the search's time range
should also be 20 minutes (-20m).
142
Manage the priority of concurrently scheduledsearches
Depending on your Splunk Enterprise deployment, you might be able to run only
one scheduled search at a time. In this case, when you schedule multiple
searches to run at approximately the same time, the search scheduler ensures
that all scheduled searches run consecutively for the period of time over which
they gather data.
However, you might have cases where you need certain searches to run ahead
of others. This is to ensure that the searches obtain current data or to ensure that
there are no gaps in data collection.
143
Best practices for scheduled alerts
Coordinating the alert's schedule with the search time range prevents situations
where event data is evaluated twice by the search. This can happen if the search
time range exceeds the search schedule, resulting in overlapping event data
sets.
In cases where the search time range is shorter than the time range for the
scheduled alert, an event might never be evaluated.
This example shows how to configure an alert that builds 30 minutes of delay into
the alert schedule. Both the search time range and the alert schedule span one
hour, so there are no event data overlaps or gaps.
144
The alert runs every hour at the half hour. It collects an hour's worth of event
data, beginning an hour and a half before the search runs. When the scheduled
search kicks off at a designated time, such as 3:30 pm, it collects the event data
that was indexed from 2:00 pm to 3:00 pm.
1. From the Search Page, create a search and select Save As > Alert.
2. In the Save As Alert dialog, specify the following to schedule the alert:
145
Set the triggering conditions when you set values for the Trigger condition field
in the Save As Alert dialog box, as described in the following subtopics.
Basic conditional alert
• Number of results
• Number of hosts
• Number of sources
The alert triggers when the number of hosts in the results rises by a count of
more than 12.
1. From the Search Page, create a search and select Save As > Alert.
2. In the Save As Alert dialog box, specify the following fields to schedule
the alert:
146
Basic conditional alert for rolling-window alerts
The behavior for basic conditional alerts differs slightly for a rolling-window alert.
The alert triggers when the set condition occurs within the rolling time window of
the search.
A secondary conditional search can help reduce the incidence of false positive
alerts.
In the following example, the alert triggers when there are 10 or more log level
events that are not INFO. When the alert triggers, it sends an email with the
results of the search. The search results detail the count for each log level.
1. From the Search Page, create the following search. Specify Last 7 days
for the time period.
147
♦ Custom condition: search count > 10
4. Define an action that sends an email that includes the results of the
search.
When you configure a Send Email action that includes search results, the
email contains the results of the original base search. It does not include
the results of the custom search.
It might appear that you can get the same results if you specify instead, the
following search for the base search of a basic conditional search:
However, a basic conditional alert based on this search provides different results.
The search results contain only log level values that are greater than 10. The
results from the advanced conditional search details the count for all log levels,
but triggers only when the log levels are greater than 10.
The behavior for advanced conditional alerts differs slightly for a rolling-window
alert, which runs in real-time. For a rolling-window alert, the alert triggers when
the set condition occurs within the rolling time window of the search.
148
For the previous example, you can design a rolling-window alert with the same
base search and get similar results with the custom condition search. Set the
rolling window to a 10 minutes time span. When the real-time search returns 10
log level entries within the 10 minute time span, the alert triggers.
The per-result alert is the most basic type of alert. It runs in real-time over an
"all-time" time span. The alert triggers whenever the search returns a result.
You can create a search to retrieve events from an index. You can also use
transforming commands to return results based on processing the retrieved
events. A per-result alert triggers in both cases, when the search returns an
event or when a transforming command returns results.
149
Create rolling-window alerts
Use a rolling-window alert to monitor and evaluate events in real time within a
rolling window. The alert triggers only when it meets the trigger condition within a
specified time period.
The rolling-window alert type is in some ways a hybrid of a per-result alert and a
scheduled alert. A rolling-window alert and a per result alert both run in real-time.
But unlike the per result alert, a rolling-window alert does not trigger each time
the search returns a result. A rolling-window alert fires only when it meets
specified trigger conditions within the specified time window. This makes the alert
similar to a scheduled alert.
1. From the Search Page, create the following search. Select Last 24 Hours
for the time range:
150
4. Continue defining actions for the alert.
When you create a rolling-window alert, you specify a time span for a real-time
search window. Real-time search windows can be any number of minutes, hours,
or days. The alert monitors events as they pass through the window in real-time.
For example, you can create an alert that triggers when a login for a user fails
four times in a 10 minute period. When the alert runs, various login failure events
pass through this window. The alert triggers only when four login failures for the
same user occur within the span of the 10 minute window.
This example might appear to fail in the following scenario. A user experiences
three login failures in quick succession. After 11 minutes pass, the user has
another login failure. The alert does not trigger because the first three failures
and the fourth failure are in different time windows.
You set the triggering conditions when you set values for the Trigger condition
field in the Save As Alert dialog, as described in the following subtopics.
151
Basic conditional alert
A basic conditional alert triggers when the number of results from a search,
within a specified time window, meet, exceed, or are less than a specified
numerical value. When you create the alert, you can specify the following
conditions:
• Number of results
• Number of hosts
• Number of sources
You create a basic conditional alert for a rolling-window similarly to how you
create one for a scheduled alert.
A secondary conditional search can help reduce the incidence of false positive
alerts.
Use case
Send an email notification if there are more than five errors in a
twenty-four hour period.
152
Alert type
Scheduled
Search
Look for error events in the last twenty-four hours.
Schedule
Run the search every day at the same time. In this case, the search runs
at 10:00 A.M.
Trigger conditions
Trigger the alert action if the search has more than five results.
Alert action
Send an email notification with search result details.
1. From the Search Page, create the following search. index=_internal "
error " NOT debug source=*splunkd.log* earliest=-24h latest=now
2. Select Save As > Alert.
153
3. Specify the following values for the fields in the Save As Alert dialog box.
♦ Title: Errors in the last 24 hours
♦ Alert type: Scheduled
♦ Time Range: Run every day
♦ Schedule: At 10:00
♦ Trigger condition: Number of Results
♦ Trigger when number of results: is greater than 5.
4. Select the Send Email alert action.
5. Set the following email settings, using tokens in the Subject and Message
fields.
♦ To: email recipient
♦ Priority: Normal
♦ Subject: Too many errors alert: $name$
♦ Message: There were $job.resultCount$ errors reported on
$trigger_date$.
♦ Include: Link to Alert and Link to Results
A real-time alert searches continuously for results in real time. You can configure real-
time alerts to trigger every time there is a result or if results match the trigger conditions
within a particular time window.
Use case
Monitor for errors as they occur on a Splunk platform instance. Send an
email notification if more than five errors occur within one minute.
Alert type
Real-time
Search
Look continuously for errors on the instance.
154
Trigger conditions
Trigger the alert if there are more than five search results in one minute.
Alert action
Send an email notification.
1. From the Search Page, create the following search. index=_internal "
error " NOT debug source=*splunkd.log*
2. Select Save As > Alert.
3. Specify the following values for the alert fields.
♦ Title: Errors reported (Real-time)
♦ Alert type: Real-time
♦ Trigger condition: Number of Results
♦ Trigger if number of results: is greater than 5 in 1 minute.
4. Select the Send email alert action.
5. Specify the following email settings, using tokens in the Subject and
Message fields.
♦ To: email recipient
♦ Priority: Normal
♦ Subject: Real-time Alert: $name$
♦ Message: There were $job.resultCount$ errors.
♦ Include: Link to Alert, Link to Results, Trigger Condition, and
Trigger Time.
Accept defaults for all other options.
6. Click Save.
Throttle an alert to reduce its triggering frequency and limit alert action behavior. For
example, you can throttle an alert that generates more email notifications than you
need.
Throttle the example real-time alert. The following settings change the alert triggering
behavior so that email notifications only occur once every ten minutes.
1. From the Alerts page in the Search and Reporting app, select the alert. The
alert details page opens.
2. Next to the alert Trigger conditions, select Edit.
155
3. Select the Throttle option. Specify a 10 minute period.
4. Click Save.
When you create an alert you can use one of the available result or field count
trigger condition options. You can also specify a custom trigger condition. The
custom condition works as a secondary search on the initial results set.
Use case
Use the Triggered Alerts list to record WARNING error instances.
Alert type
Real-time
Search
Look for all errors in real-time.
Triggering condition
Check the alert search results for errors of type WARNING. Trigger the alert
action if results include any WARNING errors.
Alert action
List the alert in the Triggered Alerts page.
1. From the Search and Reporting home page, create the following search.
index=_internal source="*splunkd.log" ( log_level=ERROR OR
log_level=WARN* OR
log_level=FATAL OR log_level=CRITICAL)
156
♦ Custom Condition: search log_level=WARN* in 1 minute
4. Select the List in Triggered Alerts alert action.
5. Click Save.
157
Lab 16 Macros
Search macros are chunks of a search that you can reuse in multiple places, including
saved and ad hoc searches. Search macros can be any part of a search, such as an
eval statement or search term, and do not need to be a complete command. You can
also specify whether or not the macro field takes
any arguments.
In Settings > Advanced Search > Search macros, click "New" to create a new search
macro.
Your search macro can be any chunk of your search string or search command pipeline
that you want to re-use as part of another search.
Destination app is the name of the app you want to restrict your search macro to; by
default, your search macros are restricted to the Search app.
If your search macro takes an argument, you need to indicate this by appending the
number of arguments to the name; for example, if mymacro required two arguments, it
should be named mymacro(2). You can create multiple search macros that have the
same name but require different numbers of arguments: foo, foo(1), foo(2), etc.
Definition is the string that your search macro expands to when referenced in another
search.
If the search macro requires the user to input arguments, they are tokenized and
indicated by wrapping dollar signs around the arguments; for example, $arg1$. The
arguments values are then specified when the search macro is invoked.
158
If Eval Generated Definition? is checked, then the 'Definition' is expected to be an eval
expression that returns a string that represents the expansion of this macro.
If a macro definition includes a leading pipe character ("|"), you may not use it as
the first term in searches from the UI.
The UI does not do the macro expansion and cannot correctly identify the initial pipe to
differentiate it from a regular search term. The UI constructs the search as if the macro
name were a search term, which after expansion would cause the metadata command
to be incorrectly formed and therefore invalid.
159
Example - Combine search macros and transactions
Transactions and macro searches are a powerful combination that you can use to
simplify your transaction searches and reports. This example demonstrates how you
can use search macros to build reports based on a defined transaction.
This search takes web traffic events and breaks them into sessions, using the
"makesessions" search macro:
sourcetype=access_* | `makesessions`
This search returns a report of the number of pageviews per session for each
day:
If you wanted to build the same report, but with varying span lengths, just save it
as a search macro with an argument for the span length. Let's call this search
macro, "pageviews_per_second(1)":
Now, you can specify a span length when you run this search from the Search
app or add it to a saved search:
`pageviews_per_second(span=1h)`
160
Lab 17 Lookups
201,Created,Successful
202,Accepted,Successful
203,Non-Authoritative Information,Successful
...
2. Go back to the Search app, then select Settings > Lookups.
3. In the Lookups page, select Add new for Lookup table files.
4. In the Add new page,
• Select search for the destination app.
• Browse for the CSV file that you downloaded earlier.
• Name the lookup table http_status.
• Click Save.
Now, let's go back to the Settings > Lookups view. To do this, click on the
Lookups link in the page's breadcrumb. You can always use this to navigate
back to a previous view.
Define the lookup
1. From Settings > Lookups, select Add new for Lookup definitions.
In the Add new page:
2. Select search for the Destination app.
3. Name your lookup definition http_status.
4. Select File-based under Type.
5. Click Save.
After Splunk Enterprise saves your lookup definition, it takes you to the following
page:
Notice there are some actions you can take on your lookup definition.
Permissions lets you change the accessibility of the lookup table. You can
Disable, Clone, and Move the lookup definition to a different app. Or, you can
161
Delete the definition.
Once you define the lookup, you can use the lookup command to invoke it in a
search or you can configure the lookup to run automatically.
Set the lookup to run automatically
1. Return to the Settings > Lookups view and select Add new for Automatic
lookups.
In the Add new page:
2. Select search for the Destination app.
3. Name the lookup http_status.
4. Select http_status from the Lookup table drop down.
5. Apply the lookup to the sourcetype named access_combined.
6. Lookup input fields are the fields in our events that you want to match with the
lookup table. Here, both are named status (the CSV column name goes on the
left and the field that you want to match goes on the right):
7. Lookup output fields are the fields from the lookup table that you want to add
to your events: status_description and status_type. The CSV column name goes
on the left and the field that you want to match goes on the right.
8. Click Save.
162
Lab 18 Workflows
Here's an example of the setup for a GET link workflow action that sets off a Google search on
values of the topic field in search results:
163
In this example, we set the Label value to Google $topic$ because we have a field called
topic in our events and we want the value of topic to be included in the label for this workflow
action. For example, if the value for topic in an event is CreatefieldactionsinSplunkWeb the
field action displays as Google CreatefieldactionsinSplunkWeb in the topic field menu.
164
The Google $topic$ action URI uses the GET method to submit the topic value to Google for
a search.
You have configured your Splunk Enterprise app to extract domain names in web services logs
and specify them as a field named domain. You want to be able to search an external WHOIS
database for more information about the domains that appear.
Here's how you would set up the GET workflow action that helps you with this.
In the Workflow actions details page, set Action type to link and set Link method to get.
You then use the Label and URI fields to identify the field involved. Set a Label value of
WHOIS: $domain$. Set a URI value of http://whois.net/whois/$domain$.
• whether the link shows up in the field menu, the event menu, or both.
• whether the link opens the WHOIS search in the same window or a new one.
• restrictions for the events that display the workflow action link. You can target the
workflow action to events that have specific fields, that belong to specific event types, or
some combination of the two.
The Label field enables you to define the text that is displayed in either the field or event
workflow menu. Labels can be static or include the value of relevant fields.
4. Determine whether the workflow action applies to specific fields or event types in your data.
Use Apply only to the following fields to identify one or more fields. When you identify
fields, the workflow action only appears events that have those fields, either in their event
menu or field menus. If you leave it blank or enter an asterisk the action appears in menus
for all fields.
165
Use Apply only to the following event types to identify one or more event types. If you
identify an event type, the workflow action only appears in the event menus for events
that belong to the event type.
5. For Show action in determine whether you want the action to appear in the Event menu, the
Fields menus, or Both.
7. Under URI provide the URI for a web resource that responds to POST requests.
8. Under Open link in, determine whether the workflow action displays in the current window
or if it opens the link in a new window.
10. Under Post arguments define arguments that should be sent to web resource at the identified
URI.
These arguments are key and value combinations. On both the key and value sides of the
argument, you can use field names enclosed in dollar signs to identify the field value
from your events that should be sent over to the resource. You can define multiple
key/value arguments in one POST workflow action.
Enter the key in the first field, and the value in the second field. Click Add another field
to create an additional POST argument.
Splunk Enterprise automatically HTTP-form encodes variables that it passes in POST link
actions via URIs. This means you can include values that have spaces between words or
punctuation characters.
You have configured your Splunk Enterprise app to extract HTTP status codes from a web
service log as a field called http_status. Along with the http_status field the events typically
contain either a normal single-line description request, or a multiline python stacktrace
originating from the python process that produced an error.
You want to design a workflow action that only appears for error events where http_status is
in the 500 range. You want the workflow action to send the associated python stacktrace and the
HTTP status code to an external issue management system to generate a new bug report.
However, the issue management system only accepts POST requests to a specific endpoint.
Here's how you might set up the POST workflow action that fits your requirements:
166
Note that the first POST argument sends server error $http_status$ to a title field in the
external issue tracking system. If you select this workflow action for an event with an
167
http_staus of 500, then it opens an issue with the title server error 500 in the issue tracking
system.
The second POST argument uses the _raw field to include the multiline python stacktrace in the
description field of the new issue.
Finally, note that the workflow action has been set up so that it only applies to events belonging
to the errors_in_500_range event type. This is an event type that is only applied to events
carrying http_error values in the typical HTTP error range of 500 or greater. Events with
HTTP error codes below 500 do not display the submit error report workflow action in their
event or field menus.
168
Lab 19 Tagging
Tagging
Tags are used to label specific values of a ffield. For example, many names of servers
may not be immediately recognized, and using a tag format can help them be more
easily recognizable or distinguishable from each other.
To tag the value of a ffield, use the following steps:
Adding Tags
169
Naming Tags and Specifying Ffield Value Pairs
4. You will now see your tag listed as shown in the following screenshot:
170
5. Go back to the event list and click the > sign next to an event. You will see
details of the event open up in a way similar to that presented in the following
screenshot. You can see here that itemid=EST-14 has been tagged as ITEM14.
Now everywhere that EST-14 occurs, it will be tagged as ITEM14.
Tags enable you to search more easily and to convey meaning about the field
values. When you search tag=ITEM14, all the cases where itemid=EST-14
show up. By using tags in this manner, you can facilitate your analysis.
171
As an example of how to create an Event type, take the following steps using
the buttercupgames ile:
° Enter this into the search bar:
sourcetype="access_*" status=200 action=purchase
172
3. Click Save As | Event Type in the upper-right corner of the screen and
create a name for the event type. In this case, we have used the name success.
4. In this screenshot, when we enter buttercupgames | stats count by
eventtype, we get a count of each event type. In this case, we have only one
event type, so we get only one count in our table, but we could easily put
other event types in:
173
5. If you want to remove an event type, go to Settings | Event types, and you will
get a screen similar to what is shown in the following screenshot. Just ind the
event type you want to remove and click on Delete:
Event Types (Notice that you can Delete the one you just made.)
174