User login

Blogs

14

Oct

2011

I spent this week getting my first draft for my 520 report completed. Sent
it away on Wednesday and got some good feedback from Richard and Shane. I
plan on having a second draft done by mid next week, after the 518 test on
Tuesday.

10

Oct

2011

Fixed up the way clusters were created to the final state to make sure
that object distributions weren't being combined inappropriately and
making the transition too accepting. Ran the changes over my trace data
files to compare with the previous version - have improved the false
positive rate but at the expense of missing valid data, the machine is
just generally more strict, possibly overfitted. Now working on merging
links that are very very similar (distributions that overlap closely for
the majority of the data) but for some reason are not good enough for the
Kolmogorov-Smirnov test to merge.

While waiting for longer runs to finish I made a few more passes with
valgrind/callgrind to try to squeeze out a bit more performance. Tidied up
some memory allocations and cached some multi-level map lookups which
helped somewhat.

While waiting for those to run I wrote up more of the algorithms involved.
Trying to express them in a few ways (diagrams, pseudocode) as they can
get a bit complicated using straight prose. May need some better
terminology to differentiate between the parts.

10

Oct

2011

Further scamper tests based on treating a series of IP and ICMP fields as MDA load balancing flow IDs were carried out. The resulting warts files were analysed for load balancing nodes.

More literature papers have been read, and notes taken.

10

Oct

2011

Submitted the libtrace paper on Monday.

Finished my little study of the protocol mixes in the latest ISP data. Definitely looks like P2P usage has dropped compared with earlier this year. Some of the graphs can be found at http://www.wand.net.nz/~salcock/skynet/

Released a new version of libprotoident and libflowmanager.

Got back to working on libwandbgp. Finally managed to get it working without segfaulting, but still having problems with keeping the route table "up to date".

07

Oct

2011

MPLS/BGP test was this week and I finished off the last assignment for 514
which means I am now all done with that paper.

Writing, writing, writing for the rest of the week. My goal is to have my
first complete 520 report draft done by mid next week, so that I have a
couple weeks to make improvements.

04

Oct

2011

Started investigating what was causing some of the poor results in
matching SMTP and also giving rise to a large number of false positives.
Most of the false positives were due to SSH flows matching some poorly
clustered links that had huge variance and would accept objects with a
large range of sizes.

Spent most of my time wrestling with Weka and java to try to find another
way to cluster results if the initial EM clustering is obviously wrong.
It's amazing how bad EM is with small datasets despite the clusters being
quite apparent to the human eye, and trying to run test programs using
java was a painful experience. I've now removed almost every instance
where links with wildly varying object size distributions can be created,
except those going in to the final state which I'm working on now.

Initial runs with the more restrictive matching on a couple of traces is
looking positive, with all the SSH flows no longer being included, while
leaving matching of actual SMTP flows at about the same level.

04

Oct

2011

Libprotoident 2.0.3 has been released today.

This release adds support for 13 new protocols (including RADIUS, Akamai and Youku) and 3 new categories (Logging, Printing and Translation). It also improves the rules for some existing protocols and fixes a few bugs.

The included tools have all been updated to support analysis of IPv6 traffic and also provide more options for determining the direction of analysed packets.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.3 here!

03

Oct

2011

Made a few minor changes to the libtrace paper based on Richard's feedback. All ready to submit now.

Continued playing around with the new ISP trace set - hopefully will be able to put together some slightly more comprehensive results soon re: P2P usage in Jan vs Sep.

Also looked at traffic in the new traces that libprotoident could not identify. Managed to add quite a few new protocols to libprotoident - I now have rules for over 200 protocols!

On leave on Thursday and Friday.

30

Sep

2011

I finished off the COMP518 penetration testing assignment this week. Managed
to get root access to two of the machines. The windows XP firewall being
disabled helped a lot :-)

I also started writing the implementation section for my honours report. I
wrote up a few questions to be sent to Karen and Jaime to help out with the
evaluation section later on.

30

Sep

2011

This week I started making more scamper methods to test the effect of treating various IP fields as the flow ID in an ICMP packet.

More papers have been collected for my literature search, and more have been read and written up.