Monitoring Certificate Expiration Dates with Splunk and PowerShell

If you manage a large number of Windows IIS servers running lots of web services you probably also have to manage a large number of x.509 certificates. All of your IIS sites should be secured with an SSL cert, and many if not most of your web services probably are too. Many of these web services probably work with public keys from third party vendors. This quickly becomes tough to manages as the certificate count goes from dozens to hundreds.

Here is a PowerShell input for Splunk that will make managing all these certs easier. Run this input on your Windows servers with the Universal Forwarder and there will be no excuse for missing another certificate expiration date.

This will output in JSON format, one entry per certificate per server. Then its a simple matter of setting up Splunk to report or alert on expiration dates, by vendor, by issuer, or by whatever criteria fits your need. Here is the Splunk Universal Forwarder config that I use. Its created on all my Windows servers automatically using the Splunk REST API. It runs every 10 mins with a random offset determined at creation time. In this case it’s 7 minutes.

[powershell://Certificates]
schedule = 7-59/10 * * * *
script = . C:\Splunk\etc\apps\SplunkUniversalForwarder\bin\Splunk-Certificates.ps1
sourcetype = Certificates

This input requires Splunk version 6.3.x or greater. If you’re on an earlier version you can still use this PowerShell input but you’ll have to call it with the Script input instead of the newer PowerShell input.

Installing node.js on Windows Server 2012 R2 Core

When dealing with Windows Server Core you can still install node.js the same way you would if you had the full GUI. Just copy the node.js msi file to the server and run from the command line. You’ll get the same install prompts you would if the GUI was installed. Select the defaults if you like and away you go!

Node.js_Setup

That’s all well and good for one server. But if you’re installing node.js on Windows Server Core, you probably have more than one. Maybe you have 3 or 4 loadbalanced Core servers. Or maybe you have 30 or 40. Copying the msi file to each and clicking away at the install wizard is not the way to go in that case. When installing anything on more than one Windows 2012 R2 Server Core servers you want to use PowerShell DSC. Here’s a look at how you can install node.js via PowerShell DSC.

The first section of the script is your config section. Normally this would be stored in a separate psd1 file, but for this example I assigned the config data to the $config variable in the same file. The first node in the allNodes array is the “wildcard” node. Config data stored here applies to all the rest of the nodes in the array. The name of the MSI is included as well as the Product ID. The product ID is required. There are various ways to get the product ID of a particular MSI. On a system where Node.js is already installed you can execute the following PowerShell to find the ID.

PS C:\Users\USER001> Get-WmiObject Win32_Product | ? {$_.Name -eq "Node.js"}

IdentifyingNumber : {68EDB54E-2CFB-454E-BBF0-3E41E157E552}
Name              : Node.js
Vendor            : Node.js Foundation
Version           : 6.2.2
Caption           : Node.js

If you don’t have Node.js installed you can also get the Product ID straight from the MSI file with PowerShell if you have the Windows Installer PowerShell Module installed.

PS C:\Users\USER001> Get-MSITable -path .\Downloads\node-v6.2.2-x64.msi -Table Property

Property                     Value
--------                     -----
UpgradeCode                  {47C07A3A-42EF-4213-A85D-8F5A59077C28}
WixUIRMOption                UseRM
WIXUI_INSTALLDIR             INSTALLDIR
ALLUSERS                     1
ARPPRODUCTICON               NodeIcon
ApplicationFolderName        nodejs
DefaultUIFont                WixUI_Font_Normal
WixUI_Mode                   FeatureTree
WIXUI_EXITDIALOGOPTIONALTEXT Node.js has been successfully installed.
Manufacturer                 Node.js Foundation
ProductCode                  {68EDB54E-2CFB-454E-BBF0-3E41E157E552}
ProductLanguage              1033
ProductName                  Node.js
ProductVersion               6.2.2
ErrorDialog                  ErrorDlg
SecureCustomProperties       EARLY_IO_DETECTED;NODE_0X_DETECTED;WIX_DOWNGRADE_DETECTED;WIX_UPGRADE_DETECTED
PS C:\Users\USER001>

Here the product ID is called ProductCode. This is the value you’ll plug into your DSC configuration. The rest of the nodes in the allNodes array are the individual servers I want to install to. There are three listed in this example but it could just as easily be 30.
Then there is a configuration section that I’ve named Install_Node, where node equals all nodes where install equals true. That configuration gets executed to build the mof files and then the DSC configuration is started. Here’s what it looks like running against just one server. I’m doing just one server to make the output cleaner and easier to follow, but this will run against as many servers as you want to simultaneously.

PS C:\Users\USER001> .\installNode.ps1


    Directory: C:\Users\USER001\Install_Node


Mode                LastWriteTime     Length Name                                                                                                                                                                                                                
----                -------------     ------ ----                                                                                                                                                                                                                
-a---         6/29/2016   9:20 AM       1410 server01.mof                                                                                                                                                                                                          
VERBOSE: Perform operation 'Invoke CimMethod' with following parameters, ''methodName' = SendConfigurationApply,'className' = MSFT_DSCLocalConfigurationManager,'namespaceName' = root/Microsoft/Windows/DesiredStateConfiguration'.
VERBOSE: An LCM method call arrived from computer DSCSERVER01 with user sid S-1-5-21-759580286-1836913787-1093625069-111464.
VERBOSE: [SERVER01]: LCM:  [ Start  Set      ]
VERBOSE: [SERVER01]: LCM:  [ Start  Resource ]  [[Package]node.js]
VERBOSE: [SERVER01]: LCM:  [ Start  Test     ]  [[Package]node.js]
VERBOSE: [SERVER01]:                            [[Package]node.js] Validate-StandardArguments, Path was \\fileserver01\node-v6.2.2-x64.msi
VERBOSE: [SERVER01]:                            [[Package]node.js] The path extension was .msi
VERBOSE: [SERVER01]:                            [[Package]node.js] Parsing 68EDB54E-2CFB-454E-BBF0-3E41E157E552 as an identifyingNumber
VERBOSE: [SERVER01]:                            [[Package]node.js] Parsed 68EDB54E-2CFB-454E-BBF0-3E41E157E552 as {68EDB54E-2CFB-454E-BBF0-3E41E157E552}
VERBOSE: [SERVER01]:                            [[Package]node.js] Ensure is Present
VERBOSE: [SERVER01]:                            [[Package]node.js] product is
VERBOSE: [SERVER01]:                            [[Package]node.js] product as boolean is False
VERBOSE: [SERVER01]:                            [[Package]node.js] The package Node.js is not installed
VERBOSE: [SERVER01]: LCM:  [ End    Test     ]  [[Package]node.js]  in 0.3750 seconds.
VERBOSE: [SERVER01]: LCM:  [ Start  Set      ]  [[Package]node.js]
VERBOSE: [SERVER01]:                            [[Package]node.js] Validate-StandardArguments, Path was \\fileserver01\node-v6.2.2-x64.msi
VERBOSE: [SERVER01]:                            [[Package]node.js] The path extension was .msi
VERBOSE: [SERVER01]:                            [[Package]node.js] Parsing 68EDB54E-2CFB-454E-BBF0-3E41E157E552 as an identifyingNumber
VERBOSE: [SERVER01]:                            [[Package]node.js] Parsed 68EDB54E-2CFB-454E-BBF0-3E41E157E552 as {68EDB54E-2CFB-454E-BBF0-3E41E157E552}
VERBOSE: [SERVER01]:                            [[Package]node.js] Ensure is Present
VERBOSE: [SERVER01]:                            [[Package]node.js] product is
VERBOSE: [SERVER01]:                            [[Package]node.js] product as boolean is False
VERBOSE: [SERVER01]:                            [[Package]node.js] The package Node.js is not installed
VERBOSE: [SERVER01]:                            [[Package]node.js] Validate-StandardArguments, Path was \\fileserver01\node-v6.2.2-x64.msi
VERBOSE: [SERVER01]:                            [[Package]node.js] The path extension was .msi
VERBOSE: [SERVER01]:                            [[Package]node.js] Parsing 68EDB54E-2CFB-454E-BBF0-3E41E157E552 as an identifyingNumber
VERBOSE: [SERVER01]:                            [[Package]node.js] Parsed 68EDB54E-2CFB-454E-BBF0-3E41E157E552 as {68EDB54E-2CFB-454E-BBF0-3E41E157E552}
VERBOSE: [SERVER01]:                            [[Package]node.js] Package configuration starting
VERBOSE: [SERVER01]:                            [[Package]node.js] Mount share to get media
VERBOSE: [SERVER01]:                            [[Package]node.js] Starting C:\Windows\system32\msiexec.exe with /i "\\fileserver01\node-v6.2.2-x64.msi" /quiet
VERBOSE: [SERVER01]:                            [[Package]node.js] Starting process C:\Windows\system32\msiexec.exe with arguments /i "\\fileserver01\node-v6.2.2-x64.msi" /quiet
VERBOSE: [SERVER01]:                            [[Package]node.js] Package has been installed
VERBOSE: [SERVER01]:                            [[Package]node.js] Package configuration finished
VERBOSE: [SERVER01]: LCM:  [ End    Set      ]  [[Package]node.js]  in 20.3910 seconds.
VERBOSE: [SERVER01]: LCM:  [ End    Resource ]  [[Package]node.js]
VERBOSE: [SERVER01]: LCM:  [ End    Set      ]    in  26.3748 seconds.
VERBOSE: Operation 'Invoke CimMethod' complete.
VERBOSE: Time taken for configuration job to complete is 22.523 seconds

And there you have it. Node.js installed on Windows Server 2012 R2 Core using DSC. In a future post I’ll show how to install iisnode the same way.

NPM and the Corporate Proxy

After installing node.js on my Windows 10 workstation at the office I tried to install some packages via npm and was greeted with the following error.

PS C:\Users\USER001> npm install handlebars -g
npm ERR! Windows_NT 10.0.10586
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "handlebars" "-g"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code ECONNREFUSED
npm ERR! errno ECONNREFUSED
npm ERR! syscall connect

npm ERR! Error: connect ECONNREFUSED 23.235.44.162:443
npm ERR!     at Object.exports._errnoException (util.js:1007:11)
npm ERR!     at exports._exceptionWithHostPort (util.js:1030:20)
npm ERR!     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1080:14)
npm ERR!  { Error: connect ECONNREFUSED 23.235.44.162:443
npm ERR!     at Object.exports._errnoException (util.js:1007:11)
npm ERR!     at exports._exceptionWithHostPort (util.js:1030:20)
npm ERR!     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1080:14)
npm ERR!   code: 'ECONNREFUSED',
npm ERR!   errno: 'ECONNREFUSED',
npm ERR!   syscall: 'connect',
npm ERR!   address: '23.235.44.162',
npm ERR!   port: 443 }
npm ERR!
npm ERR! If you are behind a proxy, please make sure that the
npm ERR! 'proxy' config is set properly.  See: 'npm help config'

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\USER001\npm-debug.log
PS C:\Users\USER001>

To set the proxy use the following commands. Be sure to change the IP address and TCP port number to match your proxy server’s IP and port.

PS C:\Users\USER001> npm config set proxy http://10.10.10.10:8080
PS C:\Users\USER001> npm config set https-proxy http://10.10.10.10:8080

But if your corporate proxy is doing SSL interception and inspection you may still have another problem. You may get this certificate error if your proxy is using a certificate issued by a non-publicly trusted CA.

PS C:\Users\USER001> npm install handlebars -g
npm ERR! Windows_NT 10.0.10586
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "handlebars" "-g"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code UNABLE_TO_GET_ISSUER_CERT_LOCALLY

npm ERR! unable to get local issuer certificate
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\USER001\npm-debug.log
PS C:\Users\USER001>

There are two ways to handle this. The easier but less secure way to handle this is to configure npm to not use strict-ssl.

PS C:\Users\USER001> npm config set strict-ssl false

But a better and more secure way to resolve this error would be to configure npm to point to a pem file that contains the all the certificates in the certificate chain that your proxy server is using. On Windows make sure to double up the backslashes.

PS C:\Users\USER001> npm config set cafile="C:\\Certificates\\CAChain.pem"

When creating the pem file make sure that the certificate that the proxy server uses is listed first, followed by one or more intermediate certificates, and end with the root CA’s certificate.

You Want Wingdings Installed Where?

Last week I received what I considered an unusual request from one of my development teams.  The request was to install some fonts on our web services servers.  According to this development group, if these fonts aren’t installed on each and every server that their service is installed on, their web service won’t work.  This was a new one on me and this request seemed odd for a couple of reasons.  Why does a web service need fonts I asked?  It turns out that this particular web service generates PDFs of various types, so it seemed reasonable that it would require fonts with which to generate the documents.  But why would you want to install them on the server, instead of bundling them with your application I asked?  Microsoft offers a way to package fonts with your applications, and that seems like the right DevOps way to do it.

Wherever possible we try to keep the apps as independent of the servers and other infrastructure as possible. That way we can move them around much easier, and upgrades to the servers are usually much easier too. There’s also a huge advantage to keeping down the number of people or teams needed to deploy and support each app. If an application can be deployed with just the developers and application support staff then great, but add network and server dependancies and watch how much more complicated deployments get. But although this group of developers liked the idea of bundling the fonts with their application, they explained that they had no time or testing budget to change direction now.  That meant that I would have no choice, I would have to manage the fonts on our 2012 core servers for them.

But our web services servers are Windows 2012 R2 core servers. These servers are built entirely by PowerShell DSC scripts.  No one ever logs into these servers to configure anything by hand.  All changes to server configuration must be made via PowerShell DSC and checked into source control.  So I checked to see if anyone had written a PowerShell DSC resource for fonts, but not surprisingly, considering the unusual nature of this request, I didn’t find one.  Then I briefly considered rolling our own, I have written many of our own custom DSC resources, but that seemed like more work than it would be worth.  After all, how often am I going to need to manage fonts on servers?  So I choose to use the PowerShell DSC shortcut resource, the Script resource.  If you ever need to get something done quickly that you probably won’t ever have to repeat, this is your resource.

Now the request I had was to install 3 fonts, one for each of our company’s brands.  So the first wag was to add the following lines to this application’s other configuration code, and repeat for each brand’s fonts. Notice that I’m checking for the font’s file name, not the actual font name. On 2012 with a GUI you would test for the full font name, but on core only the file name is returned. Not elegant, but functional.

And so I thought that I was done. But not really, because then the request came to install a few more fonts. Could we also have Calibri and wait for it… Wingdings! Yes folks, I had an official request to install Windings on all of our Web Service servers. I guess there are lots of useful symbols in Windings so it kind of made sense, at least once you accept the idea of having the server admins managing an application’s fonts on all of our web services servers. So I added a couple more PowerShell DSC Script resources to the application’s config and thought that that would be the end of it. But I was wrong.

The next request came a few days later and it was to install all of the fonts that the developers had on their Windows 7 machines, about 300 of them.  See, the developers of this application weren’t sure which fonts the application might require (kind of depended on how the end-user used the app), so they thought that we should just install all of them, and yes this included every language, every script, and even Comic Sans!!

This is about the time I resent the development team the link to packaging fonts with your applications from Microsoft.  But alas, they still weren’t willing to move in this direction.  “No time to code, no time to test”. So if I was going to be responsible for making sure hundreds of fonts are installed on various servers, I was going to have to take a different approach.  We still needed to install them with PowerShell DSC, but I couldn’t keep updating the scripts and checking in changes every time they wanted to add a font. And I couldn’t manage each and every font individually like I had been either, it wouldn’t scale.  So I decided to write a new DSC Script resource that would check a source location (a git repository) for fonts and scan the destination folder to see if they were installed or not.  If not, DSC will install them.  Here’s how this looks.

So now the developers can manage the fonts that get installed by checking them into the git repository we set up, and each time DSC runs I’ll check and see if there are any new fonts to install. No code changes on my side when they decide to add a new font. Is this a great solution? Definitely not! I really don’t like adding the application dependency to the servers instead of leaving it with the application where it belongs. But sometimes in a DevOps world, we have to make concessions to get the apps into production, and PowerShell DSC can help make that possible.

VCP5 Delta Exam

So this morning I got around to doing something I had been putting off for a while now. I finally took the VCP5-DCV Delta Recertification Exam.  Not quite the very last possible minute, but close.  There were still 4 days left to take it.  Now I’m good for 2 more years.  I have to admit to not being fully convinced of the value of renewing the VCP certification, since VMware is releasing a new version of vSphere any day now.  But I can understand the argument that its been 2 years since I last took the exam, and VMware wants to reassure employers that VCPs really do know their stuff.  So if you’re on the fence about re-certifying, just go and get it done.  The online format makes it pretty convenient and the exam content focuses on newer product features.  But better hurry, you only have 4 more days left to take it!

Congratulations!  Thank you for participating in the VMware Certified Professional - Data Center Virtualization exam.

 

UPDATE – 3/10/2015

VMware is offering a short extension for the VCP recertification deadline.  You now have until May 8th, 2015 to renew your certification.  And for those who managed to meet the March 10th deadline, a reward of 65% off the VCP6 exam.  More details are at the VMware Education and Certification Blog.

Returning All Records When Querying the Splunk REST API

In my current environment the Windows 2012 R2 server builds are completely automated via PowerShell 4 DSC. This includes the installation of IIS Web Sites and Web Applications. We use Splunk to monitor all the IIS Logs and the .Net Web Application logs, and if you don’t have the Splunk configs automated, managing them can be a bear. Fortunately you can use PowerShell to manage Splunk via the Splunk REST API. This has been working well for us, but recently I came across an issue where the REST API on some of my servers wouldn’t return all of the monitors. These queries were working fine on most of my servers, but some would not return all of the results via the REST API, even though the Splunk command line did return all the monitors.

I found out from Splunk support that when querying the REST API for installed monitors that the results are limited to 30. To get all the results add “count=-1” as a query string to the URI for the endpoint you are calling. This isn’t documented anywhere that I could find. Here’s how it looks in PowerShell when querying the local host using the default username and password.