Splunking Websphere MQ Queues and Topics

What is Websphere MQ

IBM Websphere MQ , formerly known as MQSeries , is IBM’s Message Oriented Middleware offering and has been the most widely implemented system for messaging across multiple platforms over the last couple of decades.

What is Message Oriented Middleware

From Wikipedia :

“Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems. MOM allows application modules to be distributed over heterogeneous platforms and reduces the complexity of developing applications that span multiple operating systems and network protocols. The middleware creates a distributed communications layer that insulates the application developer from the details of the various operating system and network interfaces. APIs that extend across diverse platforms and networks are typically provided by MOM.”

Where does MQ fit into the landscape

In the course of my career , I’ve architected and coded solutions across many different verticals (aviation, core banking, card payments, telco, industrial automation, utilities) and MQ has been a fundamental mainstay in the enterprise IT fabric in all of these industries, stitching these often very heterogeneous enterprise computing environments together.Ergo , the messages being sent to MQ queues and topics represent  a massive source of valuable machine data that can be pulled into Splunk to derive operational visibility into the various systems and applications that  are communicating via MQ.

Enter Splunk

So how can we tap into MQ from Splunk ? The JMS Messaging Modular Input (freely available on splunkbase) is the answer , and I blogged about this in more detail recently.

When I first developed the JMS Messaging Modular Input , it wasn’t particularly feasible to test it against every MOM system that had a JMS provider , so my testing was done against ActiveMQ with the knowledge that JMS is just an API interface and in theory the modular input should work with all JMS providers. Upon release , the emails started coming in , and much to my delight , many users were successfully using the JMS Messaging Modular Input to Splunk their MQ environments.The theory had worked.

So recently , in collaboration with Splunk Client Architect Thomas Mann , we set about building a Websphere MQ environment and hooking the JMS Messaging Modular Input into it so as to end to end test this for ourselves.

I am not an MQ administrator by any means , I am quite familiar with JMS concepts and coding , but as far as install, admin and configuration of the MQ software , this is probably what took me the most time. Once MQ was setup properly , then configuring the JMS Messaging Modular Input via the Splunk Manager UI , was very quick and simple.

If you already have MQ in your environment and have  MQ admins that know JMS concepts with respect to setting up the MQ side and configuring the client side , then you should find all of the setup steps to be quite trivial.

Setting up MQ

Prerequisites

• Websphere version 7x installed (for this example, but previous MQ versions are compatible also)

Create a Queue

Create an MQ Queue under the default queue manger.  This step is optional if you have some destination the messages are already going to.

Create the JMS Objects

Using a JNDI File context is the simplest approach, unless you want to setup a directory service to host your JMS objects.  In this step you setup  the location of the .bindings file that MQ will create for you.  The Provider URL and Factory Class will be used later in your JMS Modular Input configuration.

Configure a connection factory

In this case I called it SplunkConnectionFactory.  This name will also be used in the JMS Modular Input configuration.

Ensure that you set the Transport mode  to Client – not Bindings

Setup server name in bindings file

Right click on the SplunkConnectionFactory and open properties.  Select the Connection item on the right hand side.  Change localhost(1414) to <servername>(1414).  This is the connection info between the .bindings file and the MQ host.  If you don’t specify that, then Splunk will try to connect to MQ on the localhost , which clearly won’t work in a remote configuration.

Create a new JMS Destination

Create a new Destination under the JMS Administered Context.  This links what is published in the .bindings file with the associated queue you want to manage.

Associate the JMS destination with a MQ Queue Manager

Disable Channel Auth in MQ

This step should not be done in production.  In that scenario, work with your MQ admin to set up appropriate access in MQ 7x and later.

1. Open a terminal and navigate to <mq install dir>/bin
2. Run runmqsc
3. Enter the following command: AlTER QMGR CHLAUTH(DISABLED), then hit Return.
4. Type END to exit runmqsc

More details here

Create the Channel

This step only needs to be done if the MQ Admin doesn’t create a channel for you.  Also, this may be unnecessary if you disable Channel Auth as outlined above.

1. First thing to do is make sure your listener is running.  runmqlsr –t tcp –m qmgr –p nnnn where qmgr is the name of your queue manager and nnnn is the port number your listener is on (default is 1414).  For my configuration, the command was:  runmqlsr –t tcp –m QMgr –p 1414
2. Create the channel for Splunk: DEFINE CHANNEL(‘splunkChannel’) CHLTYPE(SVRCONN) TRPTYPE(TCP)  DESCR(‘Channel for use by splunk programs’)
3. Create a Channel Authentication rule using the IP address of the splunk indexer that will be reading the queues.  Assign it to a user (splunk is a local user created on the MQ box, non admin but in the mqm group). Run the command: SET CHLAUTH(‘splunkChannel’) TYPE(ADDRESSMAP) ADDRESS(’10.0.0.20′)  MCAUSER(‘splunk’)
4. Grant access to connect and inquire the Queue manager: SET AUTHREC OBJTYPE(QMgr) PRINCIPAL(‘splunk’) AUTHADD(CONNECT, INQ).  Be sure to replace QMgr with your queue manager name and splunk with your local username.
5. Grant access to inquire / get / put messages on the queue. SET AUTHREC PROFILE(‘SplunkQueue’) OBJTYPE(QUEUE) PRINCIPAL(‘splunk’) AUTHADD(PUT, GET, INQ, BROWSE).  SplunkQueue is the name of the queue you created.  Replace splunk with the user id specified in step 3.

More details here

Setting up Splunk

Prerequisites

• Splunk version 5x installed
• JMS Messaging Modular Input installed

Jar files

You need to copy the MQ jar files into the mod input’s lib directory at $SPLUNK_HOME/etc/apps/jms_ta/bin/lib
These are the jars I  ended up needing , note : these 4 jars are already part of  the core mod input release.

• jmsmodinput.jar
• jms.jar
• splunk.jar
• log4j-1.2.16.jar

Bindings file

MQ will create your bindings file for you and write it to the location that you specified.
If your Splunk instance is running locally to MQ , then you are good to go.
If your Splunk instance is running remote to MQ , you can just copy the bindings file to the remote Splunk host.
The directory location of the bindings file can be anywhere you like , the path gets specified as a parameter (jndi_provider_url) when you setup the mod input stanza.

Setup the Input Stanza

You can setup the JMS Modular Input stanza manually(as a stanza entry in an inputs.conf file) or via the Splunk Manager UI.

Browse to Data Inputs >> JMS Messaging

The values that you use for the setup will come from what you setup in MQ.

Optionally , you can also configure what components of the messages you wish to index, and also whether you just want to browse the messages queue rather than consuming the messages.

This is what the resulting stanza declaration that gets written to inputs.conf will look like. The items in bold are values that come from your MQ setup.

[jms://queue/SplunkQueue]
browse_mode = all
browse_queue_only = 1
durable = 0
index = jms
index_message_header = 1
index_message_properties = 1
init_mode = jndi
jms_connection_factory_name = SplunkConnectionFactory
jndi_initialcontext_factory = com.sun.jndi.fscontext.RefFSContextFactory
jndi_provider_url = file:/home/damien/MQJNDI/
sourcetype = mq
strip_newlines = 1
browse_frequency = -1
disabled = 1

Queues : browsing or consuming ?

The JMS Messaging Modular Input allows you to specify browse mode or consume mode(default).
Browsing does not remove the messages from the queue whereas consuming does(so it is slightly more invasive).
However there are issues with browsing in 2 main respects :

1. you might miss messages if they are consumed before they are browsed
2. you might get duplicate messages if you browse multiple times before the messages get consumed

So my preferred “least invasive” approach is actually to have the MQ admin setup an alias queue where a copy of all the messages you are interested in can get sent to and then the mod input can just consume from this queue without impacting any other consumers of the source queues.

Testing

Expand the Queue Managers tab on the left hand side.  Select your queue manager and expand it (QMgr in this example).  Expand Queues.  Find the queue you have linked your JMS object too.

Right click the queue and select “Put Test Message”

You can put as many messages as you like in the queue.  You will see these messages indexed if you everything is configured correctly.