Webhooks are an incredibly useful tool. Thanks to wide support across SaaS vendors, they provide a solid foundation for integrating one system with another without either system having to know anything about the other. However, as much as I love webhooks, I don’t get to use them with Splunk as much as I’d like because of some key weaknesses in Splunk’s implementation. In this blog post I’d like to talk about those shortcomings, and how I attempted to remedy them with a new app .
What are webhooks?
Webhooks are asynchronous HTTP requests sent from a system when an event happens, typically with a JSON payload including more details on what the event was. The URL the HTTP request is sent to is user-configurable. As an example, as an OpsGenie user, you can configure the product to send you an HTTP request every time an alert is created, updated, or closed. If you’re interested in learning more about webhooks after this blog post, ngrok publishes an extremely useful resource on webhooks. I highly recommend checking that out.
Why are webhooks useful?
Let’s say my organization uses two products, PagerDuty and ServiceNow. We use PagerDuty for alert notifications, and whenever a PagerDuty incident is escalated, we create a ServiceNow ticket to track the escalation. I would love to automate this process, but I don’t want to constantly poll PagerDuty for incidents and check if they’re escalated. Thankfully, PagerDuty has webhook support! So instead, I can host a simple application that listens for POST requests to a particular URL and creates a ServiceNow ticket when it receives them. Thanks to serverless solutions such as AWS Lambda or Cloudflare Workers , I don’t even necessarily need to bother myself with any of the HTTP request handling. I then configure PagerDuty to send webhooks to that URL whenever an incident is escalated. I have now automated that entire workflow. This is the power of webhooks!
Webhooks in Splunk
Splunk ships with an app called “alert_webhook” which contains an alert action for sending webhooks. This allows you to send a webhook any time a Splunk alert fires. This can be very powerful because frequently Splunk is ingesting log data from a wide variety of systems. You could effectively “build your own” webhook support for products that don’t offer their own!
However, Splunk’s webhook action has some major weaknesses that hamstring its usefulness. First, the user has no control over the payload that is sent in each request. While the payload contents do include just about everything you could possibly want to send, the format is set in stone. I’ll get back to why this is an issue.
Next, and a much bigger problem, is authentication. Splunk has absolutely zero support for authenticated webhooks. Better hope no one on the internet discovers your webhook URL!
I’m far from the first person to identify these issues, but I set out to build an app that addressed them.
Better Webhooks
Better Webhooks is the app I built to try and help out those like myself who were hoping for more fully-featured webhooks support. It stores all credentials in Splunk’s encrypted credential store so it passes Splunk Cloud’s vetting requirements. I used Splunk UI Toolkit to build an interface for adding credentials, and wrote a simple alert action to handle sending the webhooks themselves. Below are some more details on the app’s features.
Custom Payloads
It’s much more common for a SaaS product to support outbound webhooks than it is for them to support inbound ones. In my experience, most of the time we are left to write our own handlers for webhooks, or use intermediary services such as Zapier.
Nevertheless, there do exist some products which will listen on the internet for you and handle webhooks you fling at them. Unfortunately for us Splunk users, these services almost always have their own very specific payload formats if you want to use them. Better Webhooks allows you to connect Splunk alerts up to these webhook handlers.
The app allows one to specify the entire JSON payload and supports some tokens as well. If Splunk’s default payload format works for you, I support that too.
While I haven’t seen someone do this yet, I think this alert action could be used as a simple alternative to full-fledged Splunk apps. For instance creating an alert using OpsGenie’s API is as simple as POSTing a specific JSON payload to a specific URL with a valid API key header. This could be entirely done using a Better Webhook alert action!
Authentication
If one is going to try and add support for authenticating webhooks, how would they do it? webhooks.fyi documents six different methods that vendors use to secure their webhook implementations. I didn’t implement all of them for Better Webhooks, but I did implement the two I’ve seen “in the wild”:
Shared secret
This simply requires the webhook sender to include a secret that the server can check, rejecting any request that doesn’t include it. The shared secret can be passed using HTTP basic auth or a custom HTTP header. Better Webhooks supports either option.
HMAC
While a little more complicated, HMAC is a much more interesting way to secure webhooks. In addition to allowing the receiver to authenticate the request, HMAC also allows them to confidently say the payload wasn’t tampered with during transit. It does this by hashing the webhook payload with a shared secret and a timestamp (making replay attacks more difficult). Unfortunately just as there are no widely accepted methods to securing a webhook, not everyone uses the same hash function for HMAC. I’ve seen SHA256 and SHA1, so I support both.
Conclusion and a request for feedback
I hope this helped explain my rationale behind creating the Better Webhooks app. I strongly believe that with a tool like this, Splunk can unlock all sorts of potential for organizations big and small.
Additionally, while I tried my best to cover a variety of use cases, I suspect there are features I hadn’t considered that would be useful to people. If you have an authentication scheme the app doesn’t support or some other feature request, please reach out to me at [email protected] .