What happened? "Backscatter". Someone had sent off a big load of spam with my e-mail as the return address, so I got all the bounces from the misconfigured servers out there that believed I'd really sent the junk -- 15,000-20,000 of them, I think.
So... I spend the morning cleaning up this garbage. I had to disconnect my mail server from the Internet (to stop the continued flood), and disable my spam detection (SpamAssassin) because that seemed to be a bottleneck.
One of the related problems was that if placing an e-mail in my inbox failed (which may did because the system was so clogged up), that would cause ANOTHER e-mail to be sent to me notifying me of the problem... sigh.
Here's a screenshot of Thunderbird in the middle of the mess. I had already sorted through maybe half of the unwanted mails.
Anyway... bottom line, it was a crummy morning. Lessons learned:
- backscatterer.org is wonderful. This blacklist lets me simply ignore many/most misconfigured systems that want to give me backscatter spam.
- Do not, repeat, do not use a lockfile for your SpamAssassin procmail recipe. This is why mail was not getting through. SpamAssassin takes several seconds to process an e-mail, and because I had it set to use a lockfile, only one SpamAssassin instance would run at once. In other words, I could only receive ~1000 e-mails per hour on a sustained basis before some e-mails were at risk of being dropped, and in a backscatter or spam flood like this, the rate is much higher. Here is the recipe I use now:
# Send mail through SpamAssassin. Note that we do NOT use a lockfile (unlike
# many examples on the net) in order to avoid timing out delivery under
# sustained spam barrages (we do use lockfiles below to serialize the actual
# delivery into folders).
* < 262144
(Note: Yes, I should be using spamd, and I plan to, but I haven't gotten to it yet.)