Simon Fredsted

The Webhook is the Web's way to integrate completely different systems in semi-real time.

As time has passed, the Web (or more precisely, HTTP, the protocol used for requesting and fetching the Web site you're currently reading) has become the default delivery mechanism for almost anything that's transferred over the Internet.

Webhooks are used everywhere

Webhooks are used in lots of places. Here's some examples of where Webhooks can deliver value:

• Notifying an ordering system that a payment has been completed, so the order can be shipped
• Adding a customer to a Relations Management system after they sign up on a Web site
• Transmitting events from e.g. a newsletter campaign and deliver it to an analytics platform

If it's on a network, it can send Webhooks

Refrigerators, industrial control systems, lightbulbs, speakers, routers, anti-virus programs – everything is controlled via the Web these days. It's not because HTTP is perfect, fast, or fault tolerant, rather, it's because everybody speaks it. It's very easy to send and receive a HTTP call. In fact, many programming languages can do it in a single line of code.

The Web page you're viewing right now – its URL being https://simonfredsted.com/1583 – might as well have been a Webhook. When your browser sends a request to it, my Web server sends a response back containing text, images and layout information.

The only difference between this Web site and an actual Webhook is that the sender is not a browser, but some sort of automated system, and probably contains some additional data rather than just which blog post you want to see. And in response, the system would have received a bunch of raw data rather than a formatted Web document.

With a URL, you can therefore send and receive information to and from any other system, often in the form of an API, a defined set of functions (where each is a specific URL) that can act on a system.

But what if a regular API, where I contact it myself and receive the data at my leisure, isn't enough? What if I don't want to constantly ping the API for data – what if it could just let me know when something changes? As that became a growing need, the word for applications exchanging information automatically like this came to be known as a Webhook. Something happened? Ping my Webhook and I'll know!

A general method of setting them up also started taking form: subscriptions. On most systems, this involves setting these subscriptions up by – you guessed it – a personal fax to the system operator, who can then program it into the mainframe!

Sorry, just kidding. Webhooks are of course also set up via the Web, and many sites offer either a user interface for setting them up, or even an API: that way, other applications can subscribe and unsubscribe to your application as they please.

Implementation is up to the developer

Webhooks aren't a standard, but most developers implement them in the same way: When an event happens in your program that another program is subscribed to, go though each subscription and check whether it has access to view that object, and if so, send a HTTP request to the subscriber's URL.

The content of the request can then be either the actual content (mostly in JSON or XML format), or a reference to the content that can be fetched later: something happened, but call this other URL to see exactly what happened. Doing it like that can save on bandwidth if you batch requests later, for example.

Obviously, and the whole reason we're doing this: it is then up to the subscriber to carry out an action on the Webhook if it is needed.

A diagram depicting a common way to handle webhook subscriptions in a scalable, event-based manner

Did my toaster send a Webhook about my toast finishing? Better send a text via Trello to remind me there's breakfast! (Maybe I have a bad short term memory. What's it to you?)

But what happens if problems occur along the way?

Error handling is important for a robust system

If you don't take certain precautions, Webhooks are very prone to failure and lost data.

If I'm suddenly unable to send webhooks to my subscriber, it might be a good idea to re-send it later at some point. But how long should I wait? If I send it 1 second after, the subscriber is probably still down. Maybe 10 seconds or 10 minutes is better for a specific use case. Maybe they'll be completely bombarded by my retries and they'll be worse off. Maybe random retry times, or constantly increasing (e.g. exponential backoff) ones would solve the problem.

You might also not want to keep sending a Webhook an infinite amount of times. That is a long time, after all.

Speaking of time: it might be worth it to consider timeouts for your use-case. Even though you're great at implementing fast software, maybe your customers aren't, so set a timeout that's reasonable for all parties, or you'll risk delivering duplicate Webhooks to your customers when retrying (due to said timeouts), or maybe they'll never get their data.

Maybe you want to give control to your subscribers: let them view a log of the failed requests, and let them re-send specific requests at their leisure.

Webhooks should always be queued

All of these ways of increasing robustness rely on a way to queue your calls or process them asynchronously: don't send them while delivering the request that triggers it, for example. Otherwise the user would be waiting on both the system doing its thing – and the Webhook subscriber(s) responding! Again: you can't control how long the recipient takes to answer.

Queuing is also a good idea on the subscriber side. You can't be sure that those you're subscribing to will wait forever to hand over the requests; you need to answer webhooks quickly, a couple seconds should be more than plenty to queue your action and return a response. So queue your actions, too!

As they say: your webhook should do almost nothing.

How to test Webhooks

What if you want to get to started with Webhooks, but don't have a server (that's open to the Internet) set up to receive them? What if you're developing a Webhook-based system, and you just want to see whether it successfully sends out Webhook requests?

For that use case, I created Webhook.site back in March 2016 – I often needed to test webhooks, so I decided to build my own site that worked like I wanted

Later, more features have been added to Webhook.site, like transformation and redirecting, sending emails, custom workflows, and even scripting. That way, you don't have to worry about managing servers and uptime just to, say, convert a Webhook to an email (or vice versa.), send a push notification or running a SSH command on a system. You simply use Webhook.site to automate any given Webhook.

What can I use Webhooks for?

Almost every cloud service has some sort of Webhook functionality, but if you're not a developer, they're hard to use. For that, check out Webhook.site mentioned in the previous paragraph. There's also IFTTT (if this, then that) or Zapier – under the hood, they use Webhooks.

What do you use Webhooks for? Let me know in the comments!

This entry was posted on Tuesday, July 17th, 2018 at 16:41 and is filed under Articles. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.