Domain/IP blacklists - What causes it and how to prevent?
I am having problems with a client who is getting added to blacklists. A couple of weeks ago they were finding that a lot of their outgoing mail to regular contacts was being put into Junk, it was the same week that a lot of their incoming mail from regular clients was going to their Junk.
They use Office 365 hosted exchange, I opened a support ticket with Microsoft and they came back to say a new update meant if a senders domain didn't have a particular record in the DNS (possibly reverse DNS?) then it would be sent to Junk. I said how ridiculous this was but they just kept saying it was by design and the workaround is to set Junk to be pre-appended with a period and then have it delivered to the inbox whilst keeping high confidence junk as sent to junk. That seemed to work but outgoing mail was still a problem.
I discovered that their domain was listed on a couple of blacklists, we think it may have been due to a mailshot being sent to a mailing list of several thousand people. They use Mailchimp (or similar) for mailshots and they only go to people who have signed up for them, although apparently they are poorly formatted so even people expecting them may have spammed them as they may not have looked professional. I have not seen the newsletter but the web designer told me this.
I had them removed from the blacklists so the domain was clean, week later some mails bounceback and I find that the static IP issued by BT is on the RATS Dyna list. There is a problem as reverse DNS was not set up (I contacted BT and it is now set up and not a problem on RATS) and the IP was on the worst offender list. The admin at RATS added our IP as an exclusion as he understood that it was owned by BT.
All of a sudden the domain is on some blacklists again!!
I need to work out why this is happening and how to prevent it going forward. Does anyone have any experience in this? If it wasn't something simple then it would be a paid job for any trusted members here if they were able to help, this is out of my usual remit but the web team at the company don't have any ideas so it's being left to me.
The IP is clean but the domain is currently on:
ivmURI
URIBL multi
There are two main offices using this domain for email, one in the UK, one in the US. I have not set up reverse DNS in the US.
Cheers
Re: Domain/IP blacklists - What causes it and how to prevent?
Sounds like they are using email forwarding
You send to Domain > it bounces off domain and goes to something like Gmail and then customer reply's via gmail but on behalf of the domain name
So the origin of the email is not as per domain name but gmail so auto add's it to spam/junk
Re: Domain/IP blacklists - What causes it and how to prevent?
So are you saying that the contacts they send the message to are using forwarding? Would this cause our domain to go on a blacklist though?
The domain we use is set up with Office 365 and the whole organisation uses hosted exchange. We are moving to Rackspace shortly though but will still be hosted exchange. Some of the recipients were using RS exchange, I imagine most of them are using professional email though as we are talking about recipients at national supermarkets and some huge global companies too.
Or are you talking about our incoming email?
Main problem atm is the outgoing mail as most is bouncing back and is causing us some big problems, I know it's because of the blacklists but not sure why we are on them.
URIBL rejected my removal request:
Reason: URL detected in UBE/UCE to traps - expires when traffic ceases
Not sure what that means but sounds like either our domain is sending out constant emails or another domain but with our domain in the message body.
Re: Domain/IP blacklists - What causes it and how to prevent?
Some info on UBE/UCE trap
URIBL.com |
Blacklist |
This list contains domain names belonging to and used by spammers, including but not restricted to those that appear in URIs found in Unsolicited Bulk and/or Commercial Email (UBE/UCE). This list has a goal of zero False Positives. This zone rebuilds frequently as new data is added. |
Automatic (upon receipt of a spam to spamtrap) |
Until delisting requested and issue resolved |
Red List |
This list contains domains that actively show up in mail flow, are not listed on URIBL black, and are either: being monitored, very young (domain age via whois), or use whois privacy features to protect their identity. This list is automated in nature, so please use at your own risk. |
Automatic (upon receipt of a spam to spamtrap) |
Until delisting requested and issue resolved |
Grey List |
This list contains domains found in UBE/UCE, and possibly honor opt-out requests. It may include ESPs which allow customers to import their recipient lists and may have no control over the subscription methods. This list can and probably will cause False Positives depending on your definition of UBE/UCE. This zone rebuilds several times a day as necessary. |
Automatic (upon receipt of a spam to spamtrap) |
Until delisting requested and issue resolved |
Re: Domain/IP blacklists - What causes it and how to prevent?
Inclusion on a blacklist could be for any number of reasons and usually the error does not reflect
the true reasons why but is a general error thrown out.
From experience the biggest part would they are :
1: Sending out from an IP that resides within the blacklisters DUHL which means they have the IP address
listed as a dynamic IP . If the customer has a true static IP then you would need to ensure both the forward
and reverse DNS entries map and then request the blacklister update the DUHL listing. If they do have a dynamic
IP then they need to utilise the ISP's smart host / smtp relay to send the mail out from.
So as an example
customerdomain.com
In the DNS zone file they would have soemthing along the lines of
MAIL IN A 127.0.0.1
The users who control the DNS for the IP address would then need to update the
reverse zone file for 0.0.127.in-addr.arpa to include the following :
1 IN PTR mail.customerdomain.com.
This would sort out the forward and reverse DNS mappings.
2: The customer is sending out a mail shot that has what is termed honey pots within
his recipient list. These are basically Spam traps / triggers that end users sometimes
use instead of giving out there proper mail address , when the Honey Pots recieve email
and they hit a certain score then the domain / IP gets blakclisted.Only soultion is for
the customer to vet their mailing list.
3: mail being sent out has a Null Value <> or no reply to address most mail servers see this
as spam and will block / bar the domain and IP. Check mail server configuration
4: Email signatures , a lot of receiving mail servers now vet the contents of an email
before accept / reject , if the customer uses a fancy sig file or it contains a url
that has been marked as spam in the past the mail will fail, test this by sending out plain
text messages , even HTML sig files may trigger a pattern match with the spam filters.
You can run some checks on the IP here : http://mxtoolbox.com/blacklists.aspx
this will tell you what the major blacklist players see the IP as being and if it
is blocked or not.
Hope this helps
Re: Domain/IP blacklists - What causes it and how to prevent?
Thanks mate, I was using mxtoolbox and another site too. Today we are clean on mxtoolbox but I want to avoid this from happening again in the future. Does it matter that there are two main offices sending from different IPs on the DNS config? One is in the US and one in the UK and some small satellite offices with a few people working from yet another IP.
I will check the DNS settings and go from there.
Re: Domain/IP blacklists - What causes it and how to prevent?
Are all the offices sending from the same domain name ?
If so and they are all originating from different IP's then you may want to look into getting
an SPF record set for the domain and listing what IP's are allowed to send out from the domain
Something like
TXT "v=spf1 ip4:127.0.0.1 ip4:127.0.0.2 ~all"
If additional domains are being used you need to also include those as well.
Just make sure if you go down this route you set the TTL on the record to 300 seconds ( 5 minutes )
that way if you break anything you can roll it back quickly without the restraints of the standard
24 Hour DNS propagation.
Usually if you are using Office365 all the mail should route through their system though , and your
IP should be transparent with the receiving MTA.
Re: Domain/IP blacklists - What causes it and how to prevent?
Is there an idiots guide for this somewhere? Usually this doesn't fall under my remit but the web team there said they didn't have a clue about any of this so I guess it's down to me to learn and apply.
At the moment mail is sent via Outlook and the service is hosted exchange from Office 365 but the IP of the office the email is sent from is contained in the message headers.
Re: Domain/IP blacklists - What causes it and how to prevent?
In the DNS zone file is this record present ?
IN TXT "v=spf1 include:spf.protection.outlook.com ~all"
Usually when mail services via office365 are used this SPF record would cover all
the domains and IP's using the service.
You could also add in the sending IP's also like described in the post above.
SPF examples
http://www.openspf.org/FAQ/Examples
Re: Domain/IP blacklists - What causes it and how to prevent?
It turns out that there was at least one malicious redirect on their website, they use wordpress and were affected by the recent vulnerability that affected thousands of sites. Do you happen to know if this is likely to cause problems with the blacklists I mentioned? They seem to be concerned with solely spam, I only found out about this page as Rackspace were unable to have the spammy fingerprint removed from their db until it's fixed. Bloody nightmare this one, not something I should have had to take the lead on but happy to learn some new stuff.
Re: Domain/IP blacklists - What causes it and how to prevent?
The uribl.com appear to be only interested in URL blocking not IP's . If the wordpress hack caused the customers domain to be listed then as suggested when the issue is resolved and the traffic decreases the entry will be removed from there.
Is the same server used to send out emails also ?
It appears you may have 2 issues here ?
Depending how large the customers is and if they have access to other IP's the quickest solution may be to change the IP address of the mailserver and update the DNS .
Re: Domain/IP blacklists - What causes it and how to prevent?
The site is hosted with Godaddy and the mail servers are hosted with Microsoft but the WAN IP of the office is in the message headers and this was blocked. I did contact BT to get a new static IP for the office but they said they couldn't do this.
The IP should now be clean as should the domain, it looks like there are fingerprints/cached data in some recipients mailservers though so even though we are clean they are still blocking. Quite frustrating, I guess I need to wait a day or two then try to manually contact the mailservers that are still bouncing back.
I will be speaking to their web team to tidy up the DNS so we have forward/reverse DNS and specify the IPs in the TXT record of any fixed office. Thanks for your help and advice, would never have thought about some of this.
Re: Domain/IP blacklists - What causes it and how to prevent?
No Worries , glad to see you seem to be getting everything resolved , I still can understand in this day and age of the internet there is no real governance on these third party blacklisters and they are a law unto themselves. Some of them even want to charge you to remove the IP with no guarantee that it wont happen again.
Re: Domain/IP blacklists - What causes it and how to prevent?
I did read about those scams, one of the scan sites I used warn you about some blacklists. I've got to the point where we are clean, I am manually contacting some mail servers, then I will research DNS best practices and apply them. Is this used when a company is of a certain size/high mail volume? I have never had this before but most of my clients are small offices but this is a much larger one.
Cheers
Re: Domain/IP blacklists - What causes it and how to prevent?
It depends really
Usually if an end user has a true static IP address then to comply with Best Practices and satisfy most receving MTA's forward and reverse DNS entries should be configured , the SPF record is just another bolt on to satisfy the receiving MTA's .
For most case's the IP's wont even be blacklisted but flagged within a DUHL / PBL list and marked as dynamic , and most receiving MTA's dont expect to get email from a dynamic IP as thats the first alarm bell to it being spam , users of dynamic IP's should utilise the SMTP server of their ISP. If the Listing is wrong then most of the ISP's can request the list is updated to reflect the IP's new usage and mark it as static .
With Broadband speeds getting as fast as they are , and the fact most ISP's now offer true static IP's then I think most of the larger companies will start taking them up instead of paying thousands for a Dedicated connection , the only thing they may loose is someService support in the event of an outage but even those dont crop up that often anymore and are usually fixed in 24 hours .