Comments on: Spam Statistics http://www.sunclipse.org/?p=193 Now living at http://scienceblogs.com/sunclipse/ Sat, 11 Sep 2010 03:03:11 +0000 http://wordpress.org/?v=2.5 By: Blake Stacey http://www.sunclipse.org/?p=193#comment-2972 Blake Stacey Thu, 12 Jul 2007 17:20:13 +0000 http://www.sunclipse.org/?p=193#comment-2972 Next observation: the <a href="http://en.wikipedia.org/wiki/Autocorrelation" rel="nofollow">autocorrelation</a> of the spam count decreases fairly steadily, but the autocorrelation of the "ham" count drops precipitously over about 50 days and then decreases much more slowly — almost a "hockey stick" kind of curve. If you leave out the part with the "new service" (which kicks in at day 578 of 630) then the autocorrelation drops smoothly. Next observation: the autocorrelation of the spam count decreases fairly steadily, but the autocorrelation of the “ham” count drops precipitously over about 50 days and then decreases much more slowly — almost a “hockey stick” kind of curve. If you leave out the part with the “new service” (which kicks in at day 578 of 630) then the autocorrelation drops smoothly.

]]> By: Blake Stacey http://www.sunclipse.org/?p=193#comment-2961 Blake Stacey Thu, 12 Jul 2007 13:28:35 +0000 http://www.sunclipse.org/?p=193#comment-2961 Aha — "a major site started running a type of content through it that wasn’t comments, it’s 95% legit rather than the other way around, so the ham spiked because of that." Thanks. I'll post again if the Fourier transform, autocorrelation etc. analyses show anything interesting. Aha — “a major site started running a type of content through it that wasn’t comments, it’s 95% legit rather than the other way around, so the ham spiked because of that.” Thanks.

I’ll post again if the Fourier transform, autocorrelation etc. analyses show anything interesting.

]]> By: Matt http://www.sunclipse.org/?p=193#comment-2942 Matt Thu, 12 Jul 2007 03:12:47 +0000 http://www.sunclipse.org/?p=193#comment-2942 I just explained the ham spike in the comments on the Akismet blog. I just explained the ham spike in the comments on the Akismet blog.

]]> By: Blake Stacey http://www.sunclipse.org/?p=193#comment-2937 Blake Stacey Wed, 11 Jul 2007 22:56:50 +0000 http://www.sunclipse.org/?p=193#comment-2937 Hmmm, upon cursory inspection — <i>i.e.,</i> open up <a href="http://www.octave.org/" rel="nofollow">Octave</a> and try the first things I think of — the spam spectrum seems to have a peak at a frequency of about 5 days, while the "ham" spectrum has a peak at 7. I might be able to tell more, but I have to leave the office now to head to a birthday dinner. Ciao / chow! Hmmm, upon cursory inspection — i.e., open up Octave and try the first things I think of — the spam spectrum seems to have a peak at a frequency of about 5 days, while the “ham” spectrum has a peak at 7. I might be able to tell more, but I have to leave the office now to head to a birthday dinner.

Ciao / chow!

]]> By: Blake Stacey http://www.sunclipse.org/?p=193#comment-2936 Blake Stacey Wed, 11 Jul 2007 22:31:16 +0000 http://www.sunclipse.org/?p=193#comment-2936 Data <i>and</i> a Firefox extension — oh, my! Data and a Firefox extension — oh, my!

]]> By: The algorithm http://www.sunclipse.org/?p=193#comment-2935 The algorithm Wed, 11 Jul 2007 22:25:51 +0000 http://www.sunclipse.org/?p=193#comment-2935 The chart data as XML is <a href="http://akismet.com/stats/chart-data.php" rel="nofollow">here</a>. (The almost orgasmic <a href="https://addons.mozilla.org/en-US/firefox/addon/1843" rel="nofollow">Firebug</a> extension lets you see all the network traffic from a page, and lots of other goodies. Orgasmicity may vary.) I wonder what the jump is, too. New spamming technique is definitely possible, and I can't come up with a slam-dunk other idea. At first I thought Akismet got a bunch of new users suddenly (Slashdot or something), but then spam would also spike and stay higher. If, when FFT'd, the "new ham" closely follows the weekday/weekend patterns of past ham rather than those of spam, then maybe it truly is ham and Akismet lowered its false positive rate, although that's strange because it looks like the ham <i>tripled</i> overnight. Maybe some service with its own filtering started using Akismet on its "probably ham" messages? Or, quite possibly, some chart data is flat wrong or they changed how they counted something, though I can't figure what that would be. The chart data as XML is here.

(The almost orgasmic Firebug extension lets you see all the network traffic from a page, and lots of other goodies. Orgasmicity may vary.)

I wonder what the jump is, too. New spamming technique is definitely possible, and I can’t come up with a slam-dunk other idea.

At first I thought Akismet got a bunch of new users suddenly (Slashdot or something), but then spam would also spike and stay higher. If, when FFT’d, the “new ham” closely follows the weekday/weekend patterns of past ham rather than those of spam, then maybe it truly is ham and Akismet lowered its false positive rate, although that’s strange because it looks like the ham tripled overnight. Maybe some service with its own filtering started using Akismet on its “probably ham” messages? Or, quite possibly, some chart data is flat wrong or they changed how they counted something, though I can’t figure what that would be.

]]>