Developing Modular Inputs in C#: Part 2

I’m annoyed at our engineering team, but I’ll get over it. You see, just hours after I posted my first blog post on writing modular inputs in C#, the team up in Seattle released the latest edition of the C# SDK. Within that SDK is a bunch of class libraries that do a much better job than my work on the scaffolding needed to produce a modular input. I highly recommend you go over to their site and dig in to this. Within this blog post, I’m going to adjust my code to use the new scaffolding and take a look at actually running the modular input. Let’s start with the framework. Here is a starting recipe for a modular input:

using System;
using Splunk.ModularInputs;
using System.Collections.Generic;

namespace Splunk.Twitter
{
    internal class Twitter : Script
    {
        public override Scheme Scheme
        {
            get {
                throw new NotImplementedException();
            }
        }

        public static int Main(string[] args)
        {
            return Run(args);
        }

        public override void StreamEvents(InputDefinition inputDefinition)
        {
            throw new NotImplementedException();
        }
    }
}

As you can see, there isn’t much to it – we have a property that returns our Scheme. This is basically the same Scheme class that we used in part 1, but we implement it as a property now. We also need to implement a StreamEvents() method. This is the new method that is called to actually gather events. Let’s take a look at our new Scheme implementation:

        public override Scheme Scheme
        {
            get {
                return new Scheme
                {
                    Title = "Twitter",
                    Description = "Get data from twitter",
                    StreamingMode = StreamingMode.Simple,
                    Endpoint =
                    {
                        Arguments = new List {
                            new Argument {
                                Name = "username",
                                Title = "Twitter ID/Handle",
                                Description = "Your Twitter ID"
                            },
                            new Argument {
                                Name = "password",
                                Title = "Twitter Password",
                                Description = "Your Twitter Password"
                            },
                        }
                    }
                };
            }
        }

Notice that it’s pretty much the same as before – just formatted differently. I like this one better – I don’t have to parse command line arguments, serialize the XML data or understand that the Scheme is returned from a –scheme command. It just happens for me. Now, on to the meat of todays post – actually dealing with the data. I’m not going to tell you how to connect to Twitter and pull data – there are better blog posts than mine on this subject. However, let’s explore what happens when Splunk starts a modular input to receive data. Splunkd runs the modular input with no arguments, and feeds the modular input an XML document via stdin. This is captured by the Splunk C# framework, which turns it into an InputDefinition object and then calls StreamEvents(). Your StreamEvents() method should never end (unlike mine) and can access the parameters that the modular input was configured with. You will need a sample XML document to fully test this. Here is an example:

<?xml version="1.0" encoding="utf-8" ?>
<input>
  <server_host>DEN-IDX1</server_host>
  <server_uri>https://127.0.0.1:8089</server_uri>
  <session_key>123102983109283019283</session_key>
  <checkpoint_dir>C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\modinputs\twitter</checkpoint_dir>
  <configuration>
    <stanza name="twitter://aaa">
      <param name="username">ahall</param>
      <param name="password">mypwd</param>
      <param name="disabled">0</param>
      <param name="index">default</param>
    </stanza>
  </configuration>
</input>

This is actually generated from the information you enter into the inputs.conf file or through the Manager. However, we need to hand-craft this when we are testing. My StreamEvents() method looks like this:

        public override void StreamEvents(InputDefinition inputDefinition)
        {
            Console.Out.WriteLine("# stanzas = " + inputDefinition.Stanzas.Count.ToString());
            foreach (string st in inputDefinition.Stanzas.Keys) {
                Console.Out.WriteLine(st + ":");
                Console.Out.WriteLine("\tUsername = " + inputDefinition.Stanzas[st].Parameters["username"]);
                Console.Out.WriteLine("\tPassword = " + inputDefinition.Stanzas[st].Parameters["password"]);
            }
            throw new NotImplementedException();
        }

I’m still throwing the NotImplementedException(), but first I’m printing some of the data we got from the input definition. Now you can use this to configure your modular input and start gathering data. From PowerShell, I can run this with the following command:

Get-Content MyXMLFile.xml | .\Twitter.exe

There are some great examples of modular inputs out there, including modular inputs for PowerShell execution and SNMP. Modular Inputs are a powerful method of gathering hard-to-get data, and I encourage you to explore your systems like they’ve never been explored before.