The TQMcube Zone Generator

29 December, 2006

---

Spam List Generation Upgrade

We have moved list generation — including additions and removals — from a flat file database into MySQL. We will be converting the DHCP zone in the near future. The dynamic zone is comprised of ranges which have to be parsed to CIDR with a perl script (both ways) and is further complicated by the large number of exclusions.

Improvements:

1. More information is now parsed to the database for analysis including the time stamp.
2. Better historical addition/removal data for each IP.
3. Unique indexing prevents duplication and simplifies additions.
4. This simplification — combined the upgraded parsing engine — allows us to accept delivery of most spam. Doing so precludes spammers from appreciating spam traps and spam trap domains.
5. The zone list is now sorted in true network byte order. The IP addresses are sorted by integer value. (a.b.c.d becomes a×256³ + b×256² + c×256 + d).
6. An internal white list and verification processes prevent any spam parsing errors from being published to the list.
7. We are now regenerating the spam list every ten minutes to propagate adds and removals to the mirrors more quickly.