User login

Shane Alcock's Blog

13

Jun

2011

Continued trying to get a useful libprotoident comparison result using data captured live from the ISP capture point. Managed to solve some of my memory issues by reducing the amount allocated to the DAG card - can now run tests for a decent length of time without running into swap.

However, I was still finding that many of the DPI tools were performing surprisingly poorly even when working with simple HTTP flows. Managed to track this down to a rather obscure libtrace bug where the cached capture length is not reset when using a bpf filter in combination with event-based DAG capture.

While waiting for captures etc. to run, I continued working on the text for a paper on the topic. Mostly done in terms of background, methodology and evaluation techniques - just need to start putting some useful results in there.

07

Jun

2011

Continued comparing libprotoident against various DPI-based solutions. Been trying to do some useful comparisons using ISP data (which has a much more interesting variety of traffic) but have been running into a few problems -- I can't capture full payload to disk, but running all of the traffic classifiers at the same time requires more memory than the capture box currently has.

If I disable the IP-based tracking that OpenDPI and PACE use, I can reduce the memory requirements enough to run a comparison test for a decent length of time. However, the classification accuracy of those tools drops massively, especially for P2P protocols, so the IP-based tracking is clearly more important than I had initially thought.

Set up and ran some performance tests for libprotoident and the DPI tools, measuring both CPU and memory usage.

Started writing up a draft paper on libprotoident -- not sure of a venue for it yet, but it will at least be a nice summary of all my comparative test results.

30

May

2011

Ran some experiments to compare the accuracy of libprotoident with the DPI-based traffic classification tools I've managed to get my hands on. Much of this time was spent figuring out various quirks with the other tools that was causing them to perform more poorly than expected -- e.g, TIE fails to identify HTTP by default if the GET request is more than a couple of hundred bytes.

Finally managed to get everything working properly towards the end of the week and had completed a preliminary study using some full-payload Auckland traces we'd taken last year. Results were very promising: using the PACE classifications as our ground truth, only 0.9% of traffic is not correctly identified by libprotoident, compared with 1.5% for OpenDPI and 12.4% for the L7 filter module included with TIE.

23

May

2011

Arrived back at work on Tuesday afternoon and caught up with everything that had gone on in my absence. Spent the rest of the week doing little odd jobs. Fixed a couple of libtrace bugs that had come up while I was away. Created a trac for BSOD. Fixed some byte-ordering issues in libprotoident and released version 2.0.1. Filled in some of the missing documentation on the libprotoident wiki.

On Friday, I wrote a quick tool for live reporting of protocol usage using libprotoident. Thinking we could use the live stats to make some sort of pretty on-going graphs of the Waikato traffic.

19

May

2011

Libprotoident 2.0.1 has been released!

This release fixes a number of bugs in 2.0.0, as well as adding support for new application protocols and improving the rules for many existing ones.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.1 here!

18

May

2011

I have created Trac sites for both the libprotoident and BSOD projects, so it is now possible to file tickets to report bugs or request features for either of these projects through the Trac system, rather than having to contact me directly.

The Trac sites also feature wikis which I intend to use to provide more extensive documentation for these projects, e.g. explanations of the protocols supported by libprotoident. At the moment, this is a work in progress but hopefully will get fleshed out over time.

The BSOD trac: http://wand.net.nz/trac/bsod/
The libprotoident trac: http://wand.net.nz/trac/libprotoident

17

May

2011

Attended the ICT 2011 conference in Ayia Napa, Cyprus. Unfortunately,
most of the talks were a bit outside my areas of expertise and featured
way too many mathematical symbols, so I didn't learn too much that is
relevant to us. Gave my own talk in the final session of the last day to
a very small audience, but seemed to go ok and even got a few questions
(unlike most other talks I attended).

Spent the remainder of the week in Cyprus on holiday.

11

May

2011

Made a few tweaks to my ICT presentation based on feedback I got the previous Friday, mainly adding animations and more diagrams.

Finished up and submitted the libtrace paper to IMC and the inbound session paper to ATNAC.

Made a couple of changes to the libtrace build system to satisfy Debian packaging requirements.

Created a bug tracker for libprotoident and started adding documentation to the trac wiki.

Left for Cyprus on Friday.

04

May

2011

As you may have noticed, we've upgraded the WAND website. Aside from the new theme, the biggest change is the addition of blogs for each WAND member. This provides a means for us to keep the wider world up to date with the discoveries we are making as they happen. The blogs have also been tied to our weekly reporting system, so there will be weekly updates from all research staff and students at the very least. Feel free to comment on the blogs if you have anything useful to add or wish to ask questions about the work we're doing.

At this stage, the site is still somewhat of a work in progress. Now that we've migrated successfully, we'll be auditing the content to remove out-of-date information and replace it with new content that reflects what we are doing now rather than what we did several years ago. Expect to see a few changes over the next few months...

02

May

2011

Finished up the draft of the paper for ATNAC on inbound sessions.

Wrote a draft of my talk for ICT. Gave a practice run on Friday, which
received lots of helpful feedback. Will be working on incorporating that
in before I leave on Friday.

New website went up on Thursday, so I spent a fair bit of time poking at
it and finding problems for Brad to fix. Trac spam seems to be a fairly
big problem, even after adding captchas.