19 November 2009

Is your spam filter perfect?

No!  Spam filters are not perfect. Some are too weak, or you set them too weak, and a mass of spam is delivered. Some are too strong, and genuine emails, are wrongly identified as spam. And Murphy's law decrees that the one's that get mis-filed as junk are the urgent ones you need now.


And spam filters generally operate in two distinct arenas. Most of us have our spam filtered as it arrives in the inbox, under our control, and have the spam filed in a specially-named folder which we can peruse from time to time to ensure that genuine emails aren't mis-filed. Some of us have access to spam filtering on the mailserver, which means you can remove or "flag" the spam before it arrives in your inbox. Both are under your own control.


If you use BT or Yahoo for your email service, you probably have some of the best spam protection around. But sometimes it might be "too good."  The chances are, you have also experienced the problem where someone claims to have sent you an email but you believe that you have not received it.


The sender says "I definitely sent it to your correct address" and the recipient says "It definitely has not arrived." Before you get into a serious argument with your correspondent, understand that it is quite possible that you are both right!


I have experienced dozens and dozens of cases of this and in every single one, the reason is that the BT and/or Yahoo email spam filtering service has been a little over-zealous. The email has been delivered but it has been hidden from the recipient.


I am not criticising BT/Yahoo or any other ISP who tries to protect us from all the millions of pieces of junk that are sent by email. But it is not such a good idea for them to intercept and hide legitimate emails for you. OK, they don't actually hide them but in the real world, my experience is that plenty of BT/Yahoo customers don't know where or how to find the intercepted emails.


This morning I received rejections from a message I sent to 105 addresses about computer security. Admittedly, the content of my message did repeat selected parts of a "phishing" scam email but of the 105 addressees, only those sent to BT or Yahoo were rejected. That shows that the BT/Yahoo spam filter is doing a great job in protecting their customers from virus or phishing scam emails.


While it's a great idea in principle, there seem to be two flaws in the BT/Yahoo SpamGuard system, or the way it is implemented and/or publicised:

1. It creates too many "false positives"  - identifying legitimate, normal email as unacceptable for various reasons and
2. Customers don't know how to find the intercepted emails.



Although BT/Yahoo have advised their customers how to check their "bulk mail" folders, empirical evidence indicates that very many of them have never even heard of the facility and of those who have, many of them do not know how to use it.


Here's an example of how this can easily occur (and it's based on real events) when the sender sends a perfectly normal,  innocent email to someone who has never previously received an email from that sender:


A BT/Yahoo email user might be interested in holiday accommodation and click on a website email link to send an availability enquiry.
When the holiday company replies, this is the first time the company has ever emailed that BT/Yahoo user. Very often, that email reply will be sent to the "spam" or "bulk mail" folder for no other reason than that the recipient has never received an email from the holiday company before.


The problem is made much worse, in reality, because BT/Yahoo user can only see the Bulk Mail folder if they access their mailbox via webmail. They may not know how to use webmail, or even be aware that exists.

So the sender knows they have sent their reply to the right address, because they used "reply" in their email. And the recipient "knows" the reply has not arrived in their email. It can cause a lot of misunderstanding...


Further information on Yahoo's SpamGuard can be found here...