User login

Weekly Report for week ending 29 July 2011




Rewrote the main section of the SMTP state machine code responsible for
merging states and minimising the machine. It now performs a recursive
traversal of the tree and is smarter about which states are available for
merging at any point. I improved the time taken to process 30 minutes of
an ISP trace from 25 hours to 3 and a half hours using a single thread.
Almost got all the locking sorted to be able to run successfully and
consistently with multiple threads which will hopefully increase
performance further. Initial tests with small numbers of flows and two
threads showed it taking 1/3 less time again.

Also got some useful comparisons from the first few ISP traces I've been
testing against, now that the program runs significantly faster. Using
approximately 7000 SMTP flows from a 30 minute trace as training data I
can identify about 97% of the SMTP flows in the following 30 minute
period. Unfortunately there is still quite a large number of false
positives, though the majority of those are POP3 flows which are very
similar to SMTP. Will need to see how these compare when looking at data
further away in time and location and/or with better object/protocol
identification using the new libraries.