Developing Modular Inputs in C# – Part 1

One of the cool new features of Splunk 5.0 is modular inputs, and we’ve already seen some great examples of this, such as the built-in perfmon gathering modular input and the Splunk Addon for PowerShell. However, the examples that are provided in the documentation are in Python. When I started writing my own modular input, I saw that much of the process of writing a modular input is scaffolding and repeatable. Thus I set out to write an SDK that would alleviate much of the scaffolding and provide a good framework for writing modular inputs. This multi-part series will cover the same process by writing a C# version of the Twitter example from the documentation.

The first part of writing a modular input is to implement the introspection scheme. When Splunk starts up, it searches for defined modular inputs and runs each modular input with the –scheme parameter. Splunk expects an XML document back that defines the parameters and configuration of the modular input. This is the first part that I thought I could improve with some of the scaffolding. Rather than embed the XML into the program, why not produce a definition of the scheme programmatically and then serialize it with the standard C# XML Serialization library?

Let’s look at my base program:

namespace Splunk.Twitter
{
    class Twitter
    {
        static Twitter twitterApp = new Twitter();

        static void Main(string[] args)
        {
            if (args.Length > 0 && args[0].ToLower().Equals("--scheme"))
            {
                twitterApp.Scheme();
                Environment.Exit(0);
            }
            else
            {
                Console.WriteLine("ERROR Not Implemented");
                Environment.Exit(1);
            }
        }

        public Twitter()
        {
        }
}

Our program is a standard console application that looks for when Splunk feeds us the –scheme parameter and runs the Scheme() method. Our Scheme() method will construct the introspection scheme programmatically and output it to Console.Out (the Windows equivalent of stdout):

public void Scheme()
{
    Scheme s = new Scheme
    {
        Title = "Twitter",
        Description = "Get data from Twitter",
        UseExternalValidation = true,
        StreamingMode = StreamingMode.SIMPLE
    };
    s.Endpoint.Arguments.Add(new EndpointArgument
    {
       Name = "username",
       Title = "Twitter ID/Handle",
       Description = "Your Twitter ID."
    });
    s.Endpoint.Arguments.Add(new EndpointArgument
    {
        Name = "password",
        Title = "Password",
        Description = "Your Twitter password."
    });
    Console.WriteLine(s.Serialize());
}

This is all fairly basic object creation stuff. There are a couple of enumerations that are important. Most notable in this code-segment, the StreamingMode can be SIMPLE (which is a simple line-based output similar to a log file) or XML (where each event is encapsulated in XML before being transmitted to the Splunk server for indexing). We also define the endpoint. This drives the Splunk UI when defining the new data input within the Splunk Manager. In this case, the Splunk UI will ask for two parameters – a username and password.

Compile and run the Twitter.exe application with the –scheme argument and you will see the XML introspection scheme.

<?xml version="1.0" encoding="utf-16"?>
<scheme xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <title>Twitter</title>
  <description>Get data from Twitter</description>
  <use_external_validation>true</use_external_validation>
  <use_single_instance>false</use_single_instance>
  <endpoint>
    <args>
      <arg name="username">
        <title>Twitter ID/Handle</title>
        <description>Your Twitter ID.</description>
        <required_on_edit>false</required_on_edit>
        <required_on_create>false</required_on_create>
      </arg>
      <arg name="password">
        <title>Password</title>
        <description>Your Twitter password.</description>
        <required_on_edit>false</required_on_edit>
        <required_on_create>false</required_on_create>
      </arg>
    </args>
  </endpoint>
</scheme>

Compare this to the XML embedded in the Python version of the Twitter app and you will see that this version is more compliant with an XML document (something that isn’t required by Splunk), but it is otherwise identical.

Next week, we will move on the instantiation of the modular input and getting the parameters you have configured in inputs.conf parsed. Until then, you can follow my progress on github by pulling down my github repository at http://github.com/adrianhall/splunk-csharp-modinputs-sdk.