Splunk ITSI APP
Splunk IT Service Inteligence
Plan your ITSI deployment
Deploy Splunk IT Service Intelligence (ITSI) on a configured Splunk platform installation. Review the system and hardware requirements and the search head and indexer considerations before deploying IT Service Intelligence.
Preparation for deployment
Before you deploy IT Service Intelligence, perform the following steps:
- Compile a list of services, KPIs, and glass table views that you want to create.
- Compile a list of your entities. Entities are usually hosts, but can also be users, mobile devices, and so on. Entities for hosts must include, at a minimum, the IP address, host name, and designated role. For example, web, db, or app server.
- Make sure your Splunk ITSI instance includes the default admin user. Deleting or renaming this user breaks ITSI installation and operation.
- Verify your existing hardware performance using the following search query: If the query takes more than 2-5 seconds to complete, check performance in the Job Inspector to investigate the issue. This slowness might indicate your current hardware is insufficient or badly configured, or you might have a high latency dispatch that requires architecture changes.
- Confirm Splunk Enterprise version compatibility.
Available deployment architectures
You can deploy Splunk IT Service Intelligence in a single instance deployment or a distributed search deployment. Splunk IT Service Intelligence is also available in Splunk Cloud. Before you deploy Splunk IT Service Intelligence on premises, familiarize yourself with the components of a Splunk platform deployment.
Single instance deployments
For a simple and small deployment, install ITSI on a single Splunk platform instance. A single instance functions as both a search head and an indexer. Use forwarders to collect your data and send it to the single instance for parsing, storing, and searching.
You can use a single instance deployment for a lab or test environment, or a small system with one or two users running concurrent searches.
You can deploy ITSI across any distributed architecture supported by Splunk Enterprise. This includes all types of deployment topologies, from small departmental deployments using a single instance for both indexer and search head, to large enterprise deployments using several search heads, dozens of indexers, and hundreds of forwarders.
Splunk IT Service Intelligence is available as a service in Splunk Cloud. The Splunk Cloud deployment architecture varies based on data and search load. Splunk Cloud customers work with Splunk Support to set up, manage, and maintain their cloud infrastructure.
Integration with the Splunk App for Infrastructure
As of version 4.2.0, ITSI ships with Splunk App for Infrastructure (SAI). Integration with SAI is enabled by default. If you’re upgrading ITSI while SAI is already installed in your environment, the SAI version is upgraded to the most recent version.
If you’re using the Splunk Add-on for Amazon Web Services and Splunk App for Infrastructure (SAI) to monitor AWS data, don’t install ITSI version 4.4.0 or SAI 2.0.0. SAI version 2.0.0 is not compatible with the Splunk Add-on for Amazon Web Services. If you’re using SAI to monitor AWS data with the add-on, this version of ITSI and SAI provides no way to continue doing so.
The ITSI installation package includes SAI and the Splunk Add-on for Infrastructure. Consider the following installation locations when upgrading ITSI:
- On a single instance deployment, both SAI and the Splunk Add-on for Infrastructure are installed by default with ITSI on the same instance of Splunk Enterprise.
- In a distributed environment, SAI is installed with ITSI on the search head, and the Splunk Add-on for Infrastructure is installed on the indexers.
- For a list of the directories included in ITSI, see About the ITSI installation package.
The ITSI installation package also includes the vmware_ta_itsi parent directory which contains components you need to deploy VMware data collection for SAI. If you don’t want to deploy VMware data collection, remove the directory from the ITSI package. For information about requirements and installation steps for VMware data collection components, see these topics in the Install and Upgrade Splunk App for Infrastructure guide:
- VMware data collection planning and requirements
- Install VMware data collection add-ons and dependencies
Before installing, see the release notes for SAI and the Splunk Add-on for Infrastructure:
- What’s new in Splunk App for Infrastructure in the Release Notes for Splunk App for Infrastructure manual.
- Release notes for the Splunk Add-on for Infrastructure in the Use the Splunk Add-on for Infrastructure manual.
Integration with Splunk Insights for Infrastructure
You cannot directly integrate Splunk Insights for Infrastructure with ITSI. You must upgrade your Splunk Insights for Infrastructure instance to Splunk Enterprise first.
Splunk Enterprise system requirements
Splunk IT Service Intelligence requires a 64-bit OS install on all search heads and indexers.
ITSI is incompatible with Splunk Enterprise versions 7.2.0 – 7.2.3.
To prevent ITSI Event Analytics from duplicating events on Splunk Enterprise versions 7.1.x and 7.2.4 – 7.2.10, create a limits.conf file on all search heads at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add the following stanza:
If you do not plan on using Event Analytics, the workaround is not necessary.
CPU core count and RAM are critical factors in indexer and search head performance. ITSI requires minimum hardware specifications that you increase according to your needs and usage of ITSI. These specifications also apply for a single instance deployment of ITSI.
Indexing is an I/O-intensive process. The indexers require sufficient disk I/O to ingest and parse data efficiently while responding to search requests. For the latest IOPS requirements to run Splunk Enterprise, see Reference Hardware: Indexer in the Splunk Enterprise Capacity Planning Manual.
You might need to increase the hardware specifications of your own ITSI deployment above the minimum hardware requirements depending on your environment. Depending on your system configuration, refer to the mid-range or high-performance specifications for Splunk platform reference hardware.
If the number of indexer CPU cores in your deployment exceeds the minimum hardware specifications, you can implement one of the parallelization settings to improve the indexer performance for specific use cases.
Operating system requirements
For a list of supported operating systems, browsers, and file systems, see System requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual.
When installing IT Service Intelligence on Ubuntu, use Bash shell. Do not use Dash shell as it can result in defunct processes.
ITSI license requirements
ITSI requires a separate ITSI license in addition to your Splunk Enterprise license. Your Splunk representative will provide you with an appropriate ITSI license at the time of purchase. For ITSI license installation instructions, see Install a license in the Splunk Enterprise Installation and Configuration Manual.
IT Service Intelligence ships with an internal license stack called the IT Service Intelligence Internals *DO NOT COPY* stack. Because ITSI event analytics generates a large number of notable events, this internal stack ensures that you don’t pay for these generated events. The sourcetypes used to track notable events and episodes are counted on this special stack with no impact on your Splunk Enterprise license. When calculating your daily license usage, disregard this stack, as it only counts internal ITSI usage.
IT Service Intelligence requires Java 8x – 11.x to run anomaly detection and event management features. ITSI supports OpenJDK and Oracle JDK 8-11. Java installation is required on search heads only, not indexers or forwarders.
IT Service Intelligence search head considerations
IT Service Intelligence does not require a dedicated search head. However, ITSI is not supported on the same search head as Splunk Enterprise Security (ES). For scalability beyond about 200 discrete KPIs, a search head cluster is a more stable option.
Real-time searches cannot be disabled on the search head, otherwise ITSI notable event grouping stops working.
When running a search head on a virtual machine, make sure to allocate all available CPU and RAM to the search head.
Forward search head data to indexers
ITSI runs KPI searches on the search head and by default stores data in the local itsi_summary index. It is considered a best practice to forward all internal data from search heads to indexers. There are two basic search head configuration scenarios for forwarding data to indexers:
IT Service Intelligence and search head clustering
Search head clusters increase the search load on indexers. Add more indexers or allocate additional CPU cores to the indexers when implementing a search head cluster.
Search head scaling considerations for Splunk IT Service Intelligence
Consider the following guidelines when implementing a search head cluster:
Indexer clustering support
IT Service Intelligence supports both single site and multisite indexer cluster architectures.
A single site or multisite indexer cluster architecture can have one search head or one search head cluster with a running instance of ITSI. Additional single instance search heads cannot run ITSI unless specific configuration changes are made.
For a multisite indexer cluster architecture, do the following:
- Enable summary replication. See Replicated summaries in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
- Set the ITSI search head to site0 to disable search affinity. See Disable search affinity in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
ITSI works by way of KPI collection through searches against information stored within the Splunk Enterprise environment. ITSI production deployments might require additional hardware, depending on several factors, including the existing unused capacity of the environment, the number of concurrent KPI searches, the version of Splunk Enterprise in production, and other performance considerations specific to each deployment.
Planning your hardware requirements
ITSI performance depends on the ability to perform multiple fast, concurrent searches. Performance results depend on both search optimization and the capacity of your deployment to run multiple concurrent searches.
When planning your ITSI hardware requirements, consider these ITSI-specific factors that impact performance:
- Average KPI run time
- Frequency of KPIs (1, 5, or 15 minute)
- Number of entities that are being referenced per KPI
Also consider the following Splunk Enterprise factors that might impact performance:
- Average daily index volume.
- Number of concurrent users.
ITSI capacity planning
ITSI capacity planning is governed by several variables. The three key variables in determining how many indexers and search heads you need are average KPI run time, the frequency of KPIs (1, 5, or 15 minute), and the number of entities being referenced per KPI. These variables can vary significantly in real-world deployments. Contact your Splunk sales representative for specific ITSI capacity planning recommendations based on your environment.
You must consider several other variables that impact the number of indexers and search heads you need, including the number of cores on those machines, the total amount of data being indexed, and total number of concurrent users.
Indexer and search head sizing examples
The following examples show roughly the number of indexers and search heads required to run the specified number of KPIs. These numbers are for example purposes only and vary based on your environment.
The following variables are fixed for each of the following examples:
- 5-minute KPIs
- 12 cores per search head and indexer
- Environment dedicated to ITSI alone
- Splunk Enterprise version 6.6 or later
- Use of “entity” refers to entities stored in the KV store and in the examples is a per-KPI measure, not the total number of entities in the system. If simple entity splits are done for KPIs and are not based on entities in a KV store, but extracted fields in Splunk searches, they need not be considered entities.
- 1 indexer required per 100 GB indexed
ITSI compatibility with other apps
Do not install ITSI and Splunk Enterprise Security on the same search head or search head cluster. With the exception of Enterprise Security, you can deploy ITSI on Splunk Enterprise instances with other Splunk apps.
For ITSI compatibility with all related apps and add-ons, see ITSI compatibility with other apps and add-ons.
For a comprehensive evaluation of your environment, consult Splunk Professional Services or your support representative.
SPLUNK IT SERVICE INTELLIGENCE
Transform operations with AI powered by machine learning
- Get ahead of outages with predictive health scores and KPI predictions
- Reduce MTTI and MTTR by decreasing event noise and getting to root cause faster
- Visualize and cross correlate with insights across business and IT services, applications, and infrastructure
- Deploy at scale — and in days not months — with a platform designed for on-prem, cloud or hybrid environments
Splunk IT Service Intelligence (ITSI) is a monitoring and analytics solution that empowers IT and business professionals to predict and prevent problems before they impact revenue and customer experience. With AI powered by machine learning at its core, Splunk ITSI aggregates various data types, tracks trends based on this data, and is designed to deliver the business-critical insights and predictions you need to stay ahead of service degradations, resource constraints and system outages. Customers like TransUnion trust Splunk ITSI to create a unified view of critical IT and business services, applications and infrastructure. They rely on ITSI to predict imminent outages, highlight anomalies, detect root causes, and pinpoint areas of impact, enabling them to deliver operations and services that exceed business and customer expectations. This means fewer incidents and outages and also a reduction in incident investigation and resolution time. It also means visibility and insight into the health of services that simply can’t be achieved with siloed point solutions. Unlike legac
y platforms that silo data, don’t scale, and can’t trend and predict problems, Splunk ITSI is built on the Splunk platform, bringing disconnected data together. Splunk scales when and how you need, and is designed to deliver the data-driven insight you need to detect problems, simplify investigations, triage issues and accelerate resolutions. Splunk ITSI wrangles large amounts of log, text, wire, metric, API and even social mediaderived data that support on premises, cloud, or hybrid-supported applications and infrastructure, and then easily apply machine learning to real-time production environments in a single, accessible and configurable view.
What is Service Intelligence?
Splunk IT Service Intelligence (ITSI) is a monitoring and analytics solution that uses AI to predict and help prevent problems before they impact revenue and customer experience. Shift from reactive to predictive IT and gain insight across your IT and business services, applications and infrastructure.
Employ AI to Predict and Prevent Imminent Outages
IT organizations need to keep systems and services from going down and be able to investigate the critical issues impacting the business. But between too many siloed monitoring tools, and getting bogged down in event storms and noise, IT teams struggle with how frustratingly long it takes to investigate and resolve issues. AI powered by machine learning is the key to reliable uptime and cutting through the noise. Splunk ITSI is a monitoring and analytics solution that uses AI to help decrease the event noise and help you predict things like service degradation 30–40 minutes in advance, helping to get ahead of investigation and issue resolution across your services, applications and infrastructure in order to protect your revenue and customer experience. Predictive cause analysis and KPI prediction shows you precisely which service is likely to degrade and when, and lets you instantly drill in to see root cause.
Create a 360-Degree View for Smarter Troubleshooting and Monitoring
IT organizations must move beyond just maintaining uptime. They must protect and preserve revenue, brand reputation, and customer satisfaction — all while reducing complexity at every level and trade friction with the business for unified insights. They need a platform that brings together data and performance metrics from any part of the business and IT, in one consumable place, so every team can move forward with the knowledge they need. With Splunk ITSI, you have the 360view you need to prioritize events, accelerate troubleshooting and understand impact immediately.
Scale When and How You Need
Data is the fuel for successful AI, but most platforms can’t collect and analyze the kinds of structured and unstructured data every organization has, nor do it in real-time and at scale. Splunk that has the ability to wrangle large amounts of data that support on premises, cloud, or hybrid-supported applications and infrastructure, and then easily apply machine learning to their real-time production environments in a single, accessible and configurable view.
Transform IT Operations with a True AIOps Platform
Artificial intelligence platforms are the accelerant for successful digital transformation initiatives. However, most IT organizations still struggle to understand how best to leverage these type of platforms and gain the most impact. Splunk ITSI is one of few platforms recognized by industry-leading analysts as a true AIOps platform, capable of applying big data, artificial intelligence, and machine learning capabilities across all primary IT operations functions and scale with respect to variety, velocity, and volume. This platform marries machine data with machine learning to help you simplify incident detection, train systems on incident severity, detect root cause more quickly, trigger procedures for specific alerts, and predict the future state of a system when a failure might occur. Investing in Splunk ITSI as your AIOps platform gives you the agility and cost reduction necessary to have a competitive edge today within both IT and the business.
Enjoy a Full-stack Monitoring Suite
Streamline troubleshooting and monitoring workflows by sending infrastructure data from Splunk App for Infrastructure (SAI) into Splunk ITSI. Get a holistic view of IT and business performance. Search and analyze across multiple layers of the IT stack and drill into the raw infrastructure log or metric for advanced troubleshooting.