Monitoring Certificate Expiration Dates with Splunk and PowerShell

If you manage a large number of Windows IIS servers running lots of web services you probably also have to manage a large number of x.509 certificates. All of your IIS sites should be secured with an SSL cert, and many if not most of your web services probably are too. Many of these web services probably work with public keys from third party vendors. This quickly becomes tough to manages as the certificate count goes from dozens to hundreds.

Here is a PowerShell input for Splunk that will make managing all these certs easier. Run this input on your Windows servers with the Universal Forwarder and there will be no excuse for missing another certificate expiration date.

This will output in JSON format, one entry per certificate per server. Then its a simple matter of setting up Splunk to report or alert on expiration dates, by vendor, by issuer, or by whatever criteria fits your need. Here is the Splunk Universal Forwarder config that I use. Its created on all my Windows servers automatically using the Splunk REST API. It runs every 10 mins with a random offset determined at creation time. In this case it’s 7 minutes.

[powershell://Certificates]
schedule = 7-59/10 * * * *
script = . C:\Splunk\etc\apps\SplunkUniversalForwarder\bin\Splunk-Certificates.ps1
sourcetype = Certificates

This input requires Splunk version 6.3.x or greater. If you’re on an earlier version you can still use this PowerShell input but you’ll have to call it with the Script input instead of the newer PowerShell input.

Returning All Records When Querying the Splunk REST API

In my current environment the Windows 2012 R2 server builds are completely automated via PowerShell 4 DSC. This includes the installation of IIS Web Sites and Web Applications. We use Splunk to monitor all the IIS Logs and the .Net Web Application logs, and if you don’t have the Splunk configs automated, managing them can be a bear. Fortunately you can use PowerShell to manage Splunk via the Splunk REST API. This has been working well for us, but recently I came across an issue where the REST API on some of my servers wouldn’t return all of the monitors. These queries were working fine on most of my servers, but some would not return all of the results via the REST API, even though the Splunk command line did return all the monitors.

I found out from Splunk support that when querying the REST API for installed monitors that the results are limited to 30. To get all the results add “count=-1” as a query string to the URI for the endpoint you are calling. This isn’t documented anywhere that I could find. Here’s how it looks in PowerShell when querying the local host using the default username and password.