I think the "securing webhooks" section is missing some critical tips that we've learned in production.
1) Resolve the DNS of the webhook URL, and compare all returned addresses from that resolution against an IP blacklist, which includes all RFC1918 addresses, EC2 instance metadata, and any other concerning addresses.
2) Even though it seems like you'd want to, do NOT blindly return an unexpected response to the person configuring the webhook. Say there was an error, what the code was, etc, but returning the response body means you basically just gave someone curl with a starting point on your network (see 1 as well)
3) Find ways to perform other validations of those webhooks. Are the URLs garbage? Are they against someone else's system? Create validation workflows that require initial pushes to the URL with a validation token to be entered back into your system, like validating an email address by clicking a link.
We wrestled with #1 (and therefore #2) for a long time. Amazing how carful you have to be. EC2 meta data is a place where a lot of services have their pants down unknowingly.
Our eventual solution? AWS Lambda. We built a simple function that receives a payload with the HTTP request to be made and the Lambda function makes the request. It serves as a sandboxed micro-proxy for all of our untrusted external HTTP calls. We give that Lambda function permission to do nothing within the AWS account. We even went to far as to place the Lambda in a dedicated AWS account to further isolate it, which prevents an admin accidentally placing the Lambda within a sensitive VPC, for example.
We still examine endpoint URLs to insure they don't belong to the internal network, but I sleep much better knowing that if something slips through the Lambda function is isolated from our internal resources and there's not too much interesting to see / probe / find.
Point 1, can be a little more tricky than it seems. At first you'll think, I'll just use a regex to match known local addresses to protect again evil callback urls like http://127.0.0.1/status.
You'll realize though you have to actually resolve hostnames, because users can just create an A record of foo.bar.com that points to 127.0.0.1.
Point 3 is spot on. I think it would be a good strategy to avoid expire dates on subscriptions. Producers could take decisions on whether or not keep sending data by monitoring responses on the consumer's target URL.
Another thing that it should be worth mentioning is that some services batch notifications (e.g Facebook Messenger) so that they can send more data in a single POST request.
Ehh. I disagree with both Default Permit and Enumerating Badness--I think they have their place. If I run a club do I background check and whitelist every customer? Or to a blacklist the troublemakers? The problems cited in the article were reasonable decisions at the time, but years later grew into headaches when the use-cases changed.
Does their no Default Permit policy apply to network egress? Do I have to approve each and every application that wants to connect to the Internet? I think the leaving port 80 open because it was whitelisted is why so many things tunnel through port 80 instead of using other protocols and ports. Now how do you filter and whitelist traffic?
His example of antivirus products using Enumerating Badness is a market failing more than anything else. I'm not sure I see the alternative for a naive user. Call a specialist to investigate their use-cases and "open the system" to accommodate? Any time you want to update your tool or workflow or try something new have that specialist come out and reevaluate your system?
I understand what you're saying here. But the baseline sanity set is pretty fixed. Localhost, RFC1918, IPv6 link local, etc. I'm not advocating folks blacklist every bad actor on the internet - that obviously cannot work - but there's some simple things you can do to prevent a malicious user from configuring webhooks that attack your internal services.
There are cases where IP blacklists are pretty much the only option you have. For example, in the case of webhooks, what would you whitelist? You cannot whitelist anything that user provides without manual approval (which can be huge overhead).
Pretty much the only alternative I can think of is to query whois databases of RIRs, but you would need blacklisting there as well since they do include private IP spaces as well (ex. you would need to blacklist netname IETF-RESERVED-ADDRESS-BLOCK).
Similar problem exists with route advertisements from transit providers. They are not going to provide you a list of routes they advertise to you (since they don't get those from their customers usually), so your only option is to blacklist bogons yourself (unless you want to manually approve every single prefix out there as needed).
Yes, whitelisting makes much more sense. Github has an API that you can ask about which IPs are in their network - compare the webhook sender against that list and you're dandy. This should become a standard in webhook APIs.
Yes but that's why you build webhook for, to let people consume your content. I think DX is critical today and big companies can afford to do the heavy-lifting. As I mentioned in the Subscription Expiration paragraph I totally get Microsoft's reason to put a 72hrs expire date on subscriptions but it adds some friction on the consumer side.
1) Resolve the DNS of the webhook URL, and compare all returned addresses from that resolution against an IP blacklist, which includes all RFC1918 addresses, EC2 instance metadata, and any other concerning addresses.
2) Even though it seems like you'd want to, do NOT blindly return an unexpected response to the person configuring the webhook. Say there was an error, what the code was, etc, but returning the response body means you basically just gave someone curl with a starting point on your network (see 1 as well)
3) Find ways to perform other validations of those webhooks. Are the URLs garbage? Are they against someone else's system? Create validation workflows that require initial pushes to the URL with a validation token to be entered back into your system, like validating an email address by clicking a link.