{"id":71691,"date":"2017-07-17T13:57:00","date_gmt":"2017-07-17T13:57:00","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=71691"},"modified":"2021-08-24T13:39:23","modified_gmt":"2021-08-24T13:39:23","slug":"investigating-cause-sql-server-high-cpu-load-conditions-happen","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/databases\/sql-server\/database-administration-sql-server\/investigating-cause-sql-server-high-cpu-load-conditions-happen\/","title":{"rendered":"SQL Server High CPU: Investigating the Causes"},"content":{"rendered":"<h2>Catching them in the act<\/h2>\n<p>There are plenty of ways of finding out what processes are consuming CPU, killing your database\u2019s performance. Beginning with SQL Server 2016, it\u2019s become so much easier to monitor performance in SQL Server with Query Store, but there are plenty of techniques that are still useful to investigate SQL Server high CPU, particularly if you don\u2019t have SQL Server 2016 Query Store on the database that you\u2019re checking.\u00a0<\/p>\n<p>I have my favourite techniques for doing this: I like to use both process explorer and <strong>sp_whoisactive<\/strong> to find the queries that are taking too much CPU in SQL Server, but I like to run them at the actual moment that the heavy usage happens, even if I\u2019m actually at home relaxing. In this article, I will show you how to use both these tools and, especially, how to do it in real time when the CPU needle is \u2018hitting the red\u2019. It is like catching a close-up of a villain in the floodlights at the very moment of the crime. I am talking about a technique where you set up an event-driven solution that will be fired by a WMI event when the CPU is high, thereby avoiding pooling and jobs having to running every few seconds. When it fires, the process checks if the source of the WMI event is SQL Server and then logs in and executes the statements for you. Yes, you don\u2019t need to be in the front of your workstation; you can check later and then perform whatever corrective action needs to be done in the light of the data you\u2019ve captured.<\/p>\n<p>One of these techniques, using Process Explorer, is very nicely <a href=\"http:\/\/michaeljswart.com\/2008\/06\/identifying-high-cpu-sql-processes\/\">described here<\/a> by Michael J Swart: Note that he updated the post, explaining that now he uses sp_whoisactive. I will show how to do it using both (Michael J Swart &#8211; <a href=\"http:\/\/michaeljswart.com\/2008\/06\/identifying-high-cpu-sql-processes\/\">Identifying High CPU SQL Processes<\/a>)<\/p>\n<p>Whatever the query you prefer to use, the big question will be how to do it in real time when the problem is actually happening, and log whatever information you need, even on the unattended server. There are plenty of times you need to do this, especially if you don\u2019t have a full-time DBA or if you are running in the cloud and needs some support from the cloud provider. You can help the support Engineer by sending him the queries that are breaking your system. In AWS, this kind of service is out of scope of support, but if you have luck to find an Engineer that knows SQL Server and is willing to help you, as I was, it will, help him or her to help you to tune the queries. You just need to leave the solution and then get the CSV log with the queries.<\/p>\n<p>This is how the solution works:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-71692\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-1.png\" alt=\"SQL Server High CPU investigation flow chart\" width=\"1175\" height=\"797\" \/><\/p>\n<p>Because the PowerShell script needs to modify the registry to change the output from the Get-Counter [1], the account that will runs the PowerShell Script will need to have administrative rights; the SQL Server Agent Service account in this case.<\/p>\n<p>In the case of AWS EC2 with SQL Server ( but not RDS, since this solution does not works in RDS), the account that is installed By default is the NT Service\\SQLServerAgent, which does not have the appropriate permissions. For that we have two options :<\/p>\n<ol>\n<li>Change the SQL Server Agent Account to LocalSystem or an appropriate service account with elevated rights. <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/cc281953.aspx\">Server Configuration &#8211; Service Accounts<\/a><\/li>\n<li>Create a SQL Proxy account to run the PowerShell Script. That is what we will use.<\/li>\n<\/ol>\n<p>[1] See the following references \u2013 <a href=\"https:\/\/support.microsoft.com\/en-us\/kb\/281884\">The Process object in Performance Monitor can display Process IDs (PIDs)<\/a> and <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/windows\/desktop\/aa372175(v=vs.85).aspx\">Handling Duplicate Instance Names<\/a><\/p>\n<h2>Configuring Permissions to run the Script<\/h2>\n<p>Open SSMS :<\/p>\n<h3>Create a Credential<\/h3>\n<p>\u2018<em>Security\u2019<\/em> -&gt; \u2018<em>Credentials\u2019<\/em>-&gt;\u2019<em>New Credential\u2019<\/em> and define a name and identity for the credential<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"457\" height=\"357\" class=\"wp-image-71693\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-2.png\" \/><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"620\" height=\"297\" class=\"wp-image-71694\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-3.png\" \/><\/p>\n<h3>Create a SQL Proxy Account<\/h3>\n<p>\u2018<em>SQL Serv<\/em><strong><em>e<\/em><\/strong><em>r Agent\u2019<\/em>-&gt;\u2019<em>Proxies<\/em>\u2019-&gt;<em>\u2019Operating System(CmdExec)<\/em>\u2019-&gt;<em>\u2019New Proxy\u2019<\/em> (Yellow box)<\/p>\n<p>Type the<strong> ProxyName<\/strong> and associate this with the previous credential account created (Green box)<\/p>\n<p>Check box what the account can do -&gt; \u2018<em>Operating System(CmdExec)<\/em>\u2019 (checking <em>\u2018PowerShell\u2019<\/em> is optional because the SQL Server job that will be created is a CmdExec Job and not PowerShell job)<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1032\" height=\"728\" class=\"wp-image-71695\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-4.png\" \/><\/p>\n<h2>Create the SQL Server Agent Job to call the PowerShell Script<\/h2>\n<p>Create a path and save the <strong><em>HighCPU.ps1<\/em><\/strong>and<strong><em> ConfigHighCpu.csv<\/em><\/strong> files in this path ie: C:\\temp<\/p>\n<p>Go to \u2018<em>SSMS<\/em>\u2019-&gt;<em>\u2019SQL Server Agent\u2019<\/em> -&gt; \u2018<em>Jobs<\/em>\u2019 -&gt; \u2018<em>New Job\u2019<\/em> (Red)<\/p>\n<p>In the tab \u2018<em>General\u2019<\/em> (on the left side), type the name of the Job and click in the Step tab (Yellow box)<\/p>\n<p>Type (see the Green box):<\/p>\n<ul>\n<li>The name of the Step. ie : <em>\u2018Step 1\u2019<\/em><\/li>\n<li>In the \u2018Type\u2019 Choose \u2018<em>Operating System (CmdExec)<\/em>\u2019<\/li>\n<li>Run as &#8211; the Proxy Account previously created<\/li>\n<li>In the Command type :<\/li>\n<\/ul>\n<p><strong>PowerShell -NoProfile -ExecutionPolicy Bypass -file &#8220;c:\\temp\\HighCPU.ps1&#8221;<\/strong>, where the \u2013file parameter requires the full path of the .ps1 file<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1424\" height=\"529\" class=\"wp-image-71696\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-5.png\" \/><\/p>\n<h2>Create a SQL Server WMI Alert for the High CPU<\/h2>\n<p>The intention is to avoid having any kind of program polling the CPU. We will, instead, use WMI Events fired by a SQL Server Agent Alert. It will be fired when the CPU exceeds the threshold defined in the WQL (Windows Query Language). It does not matter at this point if SQL Server is consuming CPU. This check will be made in the PowerShell script. The WMI SQL Server Agent Alert for high CPU is :<\/p>\n<pre>SELECT * FROM __InstanceModificationEvent WITHIN 5 WHERE TargetInstance ISA Win32_Processor' AND TargetInstance.loadpercentage &gt; 50<\/pre>\n<p>The statement <em>TargetInstance.loadpercentage &gt; 50<\/em> identifies the threshold for the CPU. In the example the alert will be fired with anything more than 50% of CPU in Total.<\/p>\n<p>To register the event :<\/p>\n<ul>\n<li>\u2018<em>SSMS\u2019<\/em>-&gt; \u2018<em>SQL Server Agent\u2019<\/em>-&gt;<em>\u2019Alerts\u2019<\/em>-&gt;(Right Click)-&gt;<em>\u2019New Alert\u2026<\/em>\u2019 (see the Red box)<\/li>\n<li>Add the name of the Alert and, in the<em> \u2018Type\u2019<\/em> combo box, select \u2018<em>WMI event alert\u2019 <\/em>(see the Green box)<\/li>\n<li>In the<em> \u2018NameSpace\u2019<\/em> type <code>\\\\.\\root\\cimv2<\/code> (Yellow box)<\/li>\n<li>In the Query Type the WQL I\u2019ve already shown you : (Blue box)<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"973\" height=\"720\" class=\"wp-image-71697\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-6.png\" \/><\/p>\n<ul>\n<li>In the \u2018<em>Response<\/em>\u2019 Tab, check the box \u2018<em>Execute Job\u2019<\/em> and select the SQL Server Agent Job previously created<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1013\" height=\"425\" class=\"wp-image-71698\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-7.png\" \/><\/p>\n<h2>Set up the Configure File<\/h2>\n<p>In the alert, we are setting up the event to be fired by total CPU so this configuration file will define the CPU threshold for the SQL Server process and each SQL Server thread from this process. Just open the <strong>ConfigHighCPU.csv<\/strong> file located at the same folder as the .ps1 file and provide:<\/p>\n<p><strong>PercentHighCPUTotalSQLServer<\/strong> \u2013 Defines the threshold for the total load percentage of the SQL Server process itself need to be above. IE 50 means that if the total process of the SQL Server is higher than 50% the PowerShell Script will gather the threads of this process.<\/p>\n<p><strong>PercentHighCpuPerThread<\/strong> \u2013 \u2013 Defines the threshold for the each thread of the SQL Server Process. IE 50 means if there are any thread higher than 50% the script will get the SQL Server statement and SQlL Server plan from this T-SQL<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"658\" height=\"134\" class=\"wp-image-71699\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-8.png\" \/><\/p>\n<p>All is now set up.<\/p>\n<p>When the Total CPU exceed 50%, the script will check if the SQL Server process CPU load percentage is higher than 50 (defined in the configuration file) and if it is, it will get the top 5 thread ids higher than 50% in a sample of 5 seconds and will connect into SQL Server, get the SQL Server statements related to these thread ids and log these in a CSV file called <strong><em>ReportHighCpuQueries.CSV. <\/em><\/strong> The information in this file is appended by the date and time.<\/p>\n<h2>ReportHighCpuQueries<strong>.CSV<\/strong><\/h2>\n<p>The output file called <strong><em>ReportHighCpuQueries.CSV <\/em><\/strong>will store the information :<\/p>\n<ul>\n<li><strong>TimeStamp<\/strong> : Date and Time of the gathering<\/li>\n<li><strong>ProcessID<\/strong> : ProcessID of the SQL Server<\/li>\n<li><strong>ThreadId<\/strong> : Thread Id of the SQL Server Process<\/li>\n<li><strong>PercentProcessorAVG<\/strong> : Average percent for this particular thread . This value is calculated always by the last gathering.<\/li>\n<li><strong>PercentProcessorMin<\/strong> : Minimum percent for this particular thread . This value is calculated always by the last gathering.<\/li>\n<li><strong>PercentProcessorMax<\/strong>: Maximum percent for this particular thread . This value is calculated always by the last gathering.<\/li>\n<li><strong>SQLSessionID<\/strong> : Session Id in the SQL Server<\/li>\n<li><strong>SQLQueryText<\/strong> : T-SQL that is running the high thread CPU<\/li>\n<li><strong>SQLQueryPlan<\/strong> : SQL Plan of the query<\/li>\n<\/ul>\n<p>The CSV file :<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1864\" height=\"307\" class=\"wp-image-71700\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-9.png\" \/><\/p>\n<p>To check the query plan<\/p>\n<p>Copy the SQLQueryPlan cell . Just the column SQLQueryPlan<\/p>\n<p>Open SSMS and type :<\/p>\n<pre>set showplan_xml on \r\n  go\r\n  select ''<\/pre>\n<p>&#8211;2 single quotes and then paste the cell you just copied between the &#8221;<\/p>\n<p>IE :<\/p>\n<pre>Set showplan_xml on\r\n  GO\r\n  SELECT '&lt;ShowPlanXML xmlns=\"http:\/\/schemas.microsoft.com\/sqlserver\/2004\/07\/showplan\" Version=\"1.5\" Build=\"13.0.1601.5\"&gt;&lt;BatchSequence&gt;&lt;Batch&gt;&lt;Statements&gt;&lt;StmtSimple StatementText=\"dbcc checkdb\" StatementId=\"1\" StatementCompId=\"1\" StatementType=\"DBCC\" RetrievedFromCache=\"false\" \/&gt;&lt;\/Statements&gt;&lt;\/Batch&gt;&lt;\/BatchSequence&gt;&lt;\/ShowPlanXML&gt;'<\/pre>\n<p>Run the Query \u2013 F5 or CRTL + E and you have the SQL Server plan from the query with problems.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1828\" height=\"376\" class=\"wp-image-71701\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-10.png\" \/><\/p>\n<p>Click in the link generated (Green)<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1522\" height=\"771\" class=\"wp-image-71702\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-11.png\" \/><\/p>\n<p>Also all the steps for each time the script ran are logged in a file called LOG.TXT in the same path.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1505\" height=\"640\" class=\"wp-image-71703\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2017\/07\/word-image-12.png\" \/><\/p>\n<p>The PowerShell script file is included with this article and it\u2019s an optimised version of a script published in <a href=\"https:\/\/sqlscope.wordpress.com\/2012\/05\/07\/identifying-high-cpu-sql-server-thread\/\">this article<\/a>. Thanks to the author of SQLscope for the original script :<\/p>\n<p><a href=\"https:\/\/sqlscope.wordpress.com\/2012\/05\/07\/identifying-high-cpu-sql-server-thread\/\">SQLScope &#8211; Identifying high CPU SQL Server threads<\/a><\/p>\n<p>In the case of <strong>sp_whoisactive<\/strong>, its even easier. You just need to setup the WMI agent job that will call a script that runs the SP_Whosactive and output to a file something like :<\/p>\n<pre>$SQL = \u201c EXEC master..sp_whoisactive \u2026\u2026\u201d \r\n  $SQLReturn = Invoke-sqlcmd -ServerInstance $ComputerName -Database master -Query $SQL #-ErrorAction stop | \r\n  Select  ColumnIwantShow1, ColumnIwantShow2 | \r\n  Export-csv -Path \"$($PSScriptRoot)\\ReportHighCpuQueries.csv\" -Force -NoClobber -NoTypeInformation \u2013Append<\/pre>\n<h1>Conclusion<\/h1>\n<p>The busy DBA wants precision in his alerts. Baselines and details are all very well, but it is too much irrelevant information. What is better is the DBA equivalent of CCTV or a dashcam that captures the moment when things go wrong and stores the data. The way we can do this is to configure SQL Agent to use WMI and specify our alerts in WQL. Doing this is like setting a trap for catching critters. We don\u2019t want to have to sit there watching a mousetrap do we? It doesn\u2019t matter what sporadic event we\u2019re interested in because we just choose the best for the task. Then we can relax, pull on the headphones and the death metal music, and dream of Copacabana beach while WMI lies in wait to catch that elusive database problem.<\/p>\n<p>&nbsp;<\/p>\n<p><em>If you like this article, you might also like<\/em>\u00a0<a href=\"https:\/\/www.red-gate.com\/simple-talk\/blogs\/how-to-find-cpu-intensive-queries\/\">How to find CPU intensive queries<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Any DBA who is trying to find the cause of an intermittent problem such as SQL Server high CPU dreams of being able to use a query or procedure take a  snap of the relevant variables at the point when the problem occurred. Laerte takes an example of a slow-running query hogging resources to show that you can capture the queries causing high CPU utilization when a WMI alert is fired, and save the results for later inspection, whenever it happens.&hellip;<\/p>\n","protected":false},"author":221715,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[143527],"tags":[5843],"coauthors":[6819],"class_list":["post-71691","post","type-post","status-publish","format-standard","hentry","category-database-administration-sql-server","tag-sql-server-monitoring"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/71691","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/221715"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=71691"}],"version-history":[{"count":7,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/71691\/revisions"}],"predecessor-version":[{"id":89528,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/71691\/revisions\/89528"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=71691"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=71691"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=71691"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=71691"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}