User login

Blogs

21

Nov

2011

I've been back full time at WAND this week after a busy semester of uni. Have spent the week tidying up projects I've been working on a bit over the semester and performing security updates on things.

People were having problems accessing the WAND website over ipv6 so looked into this with ITS and realised new warlock didn't have a static ipv6 address up on it but in the process found a few problems ITS have with ipv6 at the moment such as attempting to access an unroutable addresses will sit in a routing loop on one of their routers. They have also asked us to review the ipv6 firewalling policy for our machines. I took this opportunity to deploy an IP Address Management (IPAM) system to inventory all our machines and what services they host in the v4 and v6 world. The IPAM I rolled (phpipam) which while looking good and supports ipv6 has some quite poor code behind it. I have created a git repo in ~bmc26/git/phpipam/ and made a number of patches that fix some bugs and add some new features that I'll look at getting into upstream at some point.

We had a problem a few weeks back of a sector on a RAID being incorrectly reallocated which almost cost us a trace file but luckily swapping the disk out with the bad sector for a good disk and rebuilding the RAID fixed that issue. I have made a small recursive md5summer that I'm running over our traces so we have up to date md5sums for every trace so we can work out if trace files ever break.

Jamie and I have started to look at what we want to do with voodoo over the summer and I've started to build a development voodoo box based on squeeze that we'll try some new things on and see how a squeeze desktop machine behaves.

21

Nov

2011

Managed to create a new model for use with the Bernaille traffic classification technique, based on an hour of ISP traffic and using PACE to determine ground truth. The model does not perform much better than the default one I tested last week, despite including a few extra protocols.

Developed a new technique for comparing the various traffic classification schemes. My main problem is that even the commercial tools are not reliable enough to act as a genuine ground truth, so it becomes difficult to evaluate the accuracy of any given approach. My new approach evaluates each tool by comparing the classifications against the results produced by each other tool in turn, treating all flows that are unknown or classified differently as failure cases. The average failure rate is then calculated across all the tools compared against to produce an estimated accuracy rating for the evaluated tool.

So far, the results produced by this comparison approach have matched my expectations (libprotoident and PACE have lower failure rates, nmap has the highest) and have also highlighted the high quality of libprotoident's classifications. Hopefully, we will continue to have good results when NAVL is added to the mix.

On that note, still waiting on Vineyard to provide me with a binary that fixes the bug I reported last week - it has been acknowledged as a bug and are in the process of testing the fix now.

14

Nov

2011

Spent some time helping settle in summer students Chris and William. I'll
be involved in their projects, supplying data etc. As part of that I
collected some sample traceroute data from a few monitors to a few
destinations to get an idea of what I can get from a warts file, and to
give Chris some data to start building small maps with. Played around a
bit with scamper to see how the doubletree implementation works and if it
would be useful to use when collecting the data (and any other ways I can
use to minimise the impact of collection). Also put together some sample
code to demonstrate how to operate on the traceroute data.

Investigated information published by APNIC about addresses
assigned/allocated for NZ use to get a good initial pool of addresses to
test to in order to generate larger topology graphs. Will have to see if
there are any other avenues that will list addresses used within New
Zealand (even if they are assigned elsewhere), thinking some of the public
looking glasses may be useful here.

14

Nov

2011

Started work by reading and attempting to understand the NetMapJs library source code (created by Joel). This proved to be a challenge as there were bits and pieces of javascript syntax that I have not encountered before. I then played around with the JSON structure (that NetMapJs takes in) to create my own nodes and to better understand the way NetMapJs works.

I finished off the week by mucking around with some RRD files to get an understanding of the RRDTool. I also created some php scripts to fetch data from the aforementioned files, but it turns out the RRD-php extension support was not installed.

14

Nov

2011

I wrote programmes in java that convert a text file of traceroutes to graph formats for various graphing programmes, then played around with them with a small set of data.

I started rewriting them in C, so they can access the warts files directly.

14

Nov

2011

Daily scamper analysis of 27 addresses of paths that contained per flow load balancing based on ICMP ID was carried out. Seven scamper tests were used including UDP and TCP source port analysis. It was desired to observe the changing profile of paths that load balance on ICMP ID.

The IP header length MDA test was extended to UDP and TCP. The header length was varied using IP options and presence of per flow load balancing was determined.

14

Nov

2011

Started collating together the results of my analysis of dark and sleeper traffic in the ISP traces. It's not finished yet, but the results I have so far can be viewed at http://www.wand.net.nz/~salcock/sleepers/

CCR rejected my libprotoident paper, primarily due to a reviewer stating that we had not compared against the "state of the art" described in a paper from 2006 (http://www-rp.lip6.fr/site_npa/site_rp/_publications/737-conextFinal.pdf). This particular technique requires no packet payload, but is only able to identify 10 different TCP application protocols (although I can supposedly create new models for other TCP applications).

I tested the default models against some ISP traffic and found that it performed much better than I had expected, but was still less accurate than the weakest of the OSS DPI techniques. Their failure rate (in terms of misclassified bytes) was 24%, compared with 4.5% for libprotoident.

Started integrating Vineyard's NAVL library into my traffic classification evaluation tool. Started out OK, but ran into a few problems with not being able to force NAVL to expire internal entries for UDP flows when I have decided the flow has ended. This creates a problem if the 5-tuple reappears later, as NAVL returns an error when I try to create a new NAVL connection for that flow because NAVL believes the flow already exists. I've filed a support request, so hopefully I'll get some sort of solution in the next day or two.

Continued integrating Simon's OSPF code into libtrace.

08

Nov

2011

Short week back due to travel. Most of the week was spent catching up on
emails and things that needed to be done while I was away (weathermap
updates, etc). Spent some time looking at the way scamper was packaged for
AMP machines and checking that it would work fine for an upcoming topology
collection project, as well as investigating NZ IPv6 documentation
(community best practices) to see if there were any clues that might make
it easier to perform measurement to addresses that are actually in use.

07

Nov

2011

Started looking into the traffic sent to "sleeper" hosts, i.e. IP addresses that have been active but are now inactive. Still putting together the initial results, but there is definitely a difference between the traffic observed heading to "dark" hosts vs the traffic observed heading to sleepers.

During the sleeper analysis, I've been able to improve a few of the libprotoident rules to correctly match more of the traffic I've been looking at.

Began integrating Simon's OSPF parsing code into libtrace. Been slightly trickier than I had anticipated due to major differences between OSPFv2 (which Simon's code parses) and OSPFv3 (which we may want to parse in future).

Had a brief phone meeting with Vineyard Networks. They've agreed to give us access to their NAVL library for evaluation.

04

Nov

2011

Ran UDP-sport MDA and ICMP IPHL MDA that uses the IP stream identifier option, together to see if the same paths were traversed and to study per-flow load balancing.

Ran all available MDA tests on the 27 load balancing paths identified by the ICMP ID test out of 50000 paths.

Updated the slides for the PhD conference.