Create your own SPF Macro DNS Host - Auto SPF Flattening aka Compression

 I have been reviewing different options for SPF records recently and came across a few interesting articles. One in particular that caught my attention was:

An SPF DNS Server
https://blog.jverkamp.com/2020/07/16/an-spf-dns-server/

Essentially, JP builds an SPF hosting macro server using python and twisted

This solution appears to work well; however is definitely 'roll your own'.

As an email specialist I'm familiar with RBL (Realtime Block List) and how they leverage DNS to get a quick response to whether an IP should be blocked and whether an MTA should accept a message sent from it. It was during my investigation of SPF Macro's that the similarities of how the two worked became apparent.

While there are not many (if any) open source/off the shelf SPF macro hosting solutions. I began looking at things from the point of view 'building your own RBL' and working backwards for the SPF record use case. The concept of SPF is like RBL in reverse, i.e. a permit list.

Thankfully hosting your own RBL was a much more widely discussed topic and I came across rbldnsd https://rbldnsd.io/ which can be used to host your own RBL. 

According to their website "rbldnsd is a small and fast DNS daemon which is especially made to serve DNSBL zones. rbldnsd is extremely fast - it outperforms both bind and djbdns greatly. It has very small memory footprint."

I decided to test and investigate further, using the RBL host to deliver DIY SPF Macro hosting. So I stood up a docker container of rbldnsd using a 7 year old image (which is fine, as the project seems to have stopped ~2015):

https://github.com/kurthuwig/docker-rbldnsd

All I needed to do was :

1. Port forward UDP 53 through my firewall to the docker host - this could be hosted on AWS or Azure of course.
2. Create a config file - started with the test one here : http://www.corpit.ru/mjt/rbldnsd.html
3. Run the docker container with the 'command' relative to your config file.
4. Update your DNS host (e.g. Cloudflare) to point 'NS' record for the subdomain hosting SPF _spf.<mydomain>.com to the hostname of my home connection (ns1.<mydomain.com).
5. This then allowed me to perform NS lookups on 5.129.10.10.<mydomain>.com._spf.<mydomain>.com to get a response - any response for SPF is considered a 'pass'. e.g. if the server responds with 127.0.0.2
6. I then went a step further and configured my 'TXT' record on the sending domain for SPF to:
"v=spf1 exists:%{ir}.%{d}._spf.<mydomain>.com -all"

"v=spf1 include:%{ir}.%{d}._spf.<mydomain>.com -all"

 EDIT: by using SPF macro with include and rbldnsd it allows you to only leverage the TXT record and not the A record; this allows you to responds with 'v=spf1 -all' or 'v=spf1 exists:%{ir}.%{d}.salesforce.com -all' to reference another macro lookup. The advantage of this is that using exists: looks for whether an 'A' record is present or not and doesnt matter what it is.

NOTE: the 'i' in the  %{ir} refers to IP address e.g. 10.10.129.5, however rbldnsd responds to the reverse (5.129.10.10) so the 'r' in %{ir} reverses the IP to cater for this. The %{d} is optional and could be useful where you hosting multiple domains - 'd' is the sending envelope domain used for the SPF check. so if my domain  was microsoft.com and the sending IP was 10.10.129.5 the lookup that would be checked would be:

5.129.10.10.microsoft.com._spf.example.com  e.g.

nslookup 5.129.10.10.microsoft.com._spf.example.com. which if listed in SPF should recieve an 'A' record response of 127.0.0.2 as an example.

Following this I was able to test my SPF record using https://www.kitterman.com/spf/validate.html which confirmed it passed:

In terms of making this 'production' worthy, I will explore whether it is possible to replicate or leverage cloudflare or route53(aws) as the first DNS server that then performs a lookup on my rblsdnsd application hosted at home - they would also ideally cache the IP for 1 hour or so, to reduce the hit and criticality of my home connection.

I have also since discovered dnsdist (by powerdns) - this is an excellent load balancer for DNS and provides protection to the rblsdnsd daemon. I suspect my production implementation of this will be a Debian server in Amazon AWS with rbldnsd listening on port 5300 and dnsdist in front of it - the performance of dnsdist is very impressive and would allow multiple rblsdnd behind it, split between AWS zones.

Watch this space; I plan to put together technical step by step on this.

UPDATE: 6th Sept 2022 - I have put this into practice and created 2 docker containers. 1 hosting RBLDNSD, and 1 hosting a python SPF resolver script: https://github.com/smck83/expurgate

See sample Expurgate generated rbldnsd config file for SPF Macro hosting here spf.protection.outlook.com