User login

Blogs

10

Apr

2017

Continued delving into the unknown traffic on the campus network. Had a mix of frustrating days and successful days -- one protocol (N2Ping) took nearly two days to track down but I got there in the end. 8 new protocols added to libprotoident this week again, so we're starting to get close to 400 supported protocols in libprotoident.

Another week of refinement on the FSM code. Most of the effort has been focused on loop recognition, particularly in terms of making sure we don't ignore candidates that can be used to identify loops.

06

Apr

2017

Put the finishing touches to the AMP throughput test before building some new packages for testing. Updated the collector packages to understand the new options so that the data can be accessed/saved.

Spent some time trying to integrate VRFs into the BGP project. Routes are currently placed into a number of VRFs based on the peer, and the VRFs that routes are exported from are selected based on the peer ASN and address.

Figured out an event in Chrome that I can easily hook into to determine when a Youtube video has ended (rather than polling and hoping), and wrote code to do so. I now have a standalone program that will tell me how much time was spent in initial buffering, later stalling/rebuffering, playback, etc. Started adding a timeline of actions to show how much time was spent in each state, and when. After that will likely be the task of trying to get this running as part of AMP (which could be annoying, at least until headless mode is available in packaged versions).

06

Apr

2017

Updated the AMP throughput test to allow masquerading as an HTTP POST, so that various middleboxes interested in HTTP traffic will take interest in our flow. This required tidying up the test protocol slightly and removing the last of the in-stream signalling so that there was space for the HTTP headers. As part of this I also improved the write handling so that the select timeout will trigger properly at the end of the test duration, lowering the chances of the last write dragging out the test duration.

Continued working on the BGP project, using ExaBGP to get access to nicely formatted BGP messages. Routes updates are accepted from and sent to peers, including our own local routes and routes learnt from other peers. Also updated the simple topology generator to allow updating links via the command line in real time to help testing.

03

Apr

2017

Have been using my new daily libprotoident email to make some good progress in terms of adding new protocols to libprotoident. Another 8 protocols added this week, with 5 existing protocols improved as well.

Found a few new bugs in my FSM tandem-repeat code after running it against my full test dataset and doing an initial validation of the resulting machines. Finished up a set of slides describing (broadly) what I'm doing overall with the FSM project and how I'm going about it, i.e. suffix trees, pattern extraction, variant detection and machine building.

Started looking into a parallel RT implementation for libtrace / wdcap, with an eye towards removing the combiner bottleneck from wdcap.

27

Mar

2017

Finished implementing tandem repeat detection within my existing pattern extraction code. The initial results look promising, i.e. the code has been able to identify "write,read" as a repeat in the FTP system call log with no obvious false positives. Next job will be to repeat the machine validation and make sure that I have improved the results overall.

Wrote a libprotoident program to perform daily monitoring of unknown payload patterns on the Waikato capture point and send me an email every morning with the 25 "biggest" patterns by payload, as well as a few example flows matching each pattern. Using this data, I've already been able to add a few new patterns to libprotoident and look forward to being able to be more proactive at keeping libprotoident up to date.

23

Mar

2017

Found what looked like the perfect place in Chrome to update the headers to include Timing-Allow-Origin, but when implementing it discovered that the class I was inheriting from had redeclared all the useful bits from its parent as private and I couldn't do anything at all. Looks like I'll probably have to go up a level and duplicate the entire class to add the single line of code that I need, annoying. Had a couple of new ideas about how to get access to the appropriate javascript events that will tell me when to fetch Youtube video data, but ran out of time to test them.

Added a few simple unit tests to the BGP SDN work I'm doing to help make sure that my building blocks are all doing the correct things. Started trying to make things a bit more realistic by allowing multiple simultaneous fake BGP connections, and using the correct attributes from the feed rather than the easier ones I was previously using to test. Followed that up by installing BIRD on a VM and peering it with my code, now speaking real BGP instead of parsing MRT files.

20

Mar

2017

Finished porting the remaining libprotoident tools to be parallel-compatible. Spent a couple of days looking at unknown payload patterns in some recent Uni traffic -- unfortunately I wasn't able to make much tangible progress on identifying much of the unknown traffic.

Worked on implementing an algorithm for finding tandem repeats in strings, with the eventual aim of porting it over to work with my system call sequences. The published algorithm consists of three phases, but each of those phases has either involved looking up and implementing several other string processing algorithms (LZ-decomposition, longest common extension) or has required modifications to my existing suffix tree code (extracting a suffix array, bottom-up traversal, storing the longest child suffix in each node). Therefore, I'm about half-way through implementing the algorithm.

Moved libtrace into its own github organization to reflect that libtrace is now going to be more of a community project than a WAND project. I'll still be helping out with maintaining it for now, but now the workload can be shared amongst a group of trusted libtrace users (including people outside of WAND). This will hopefully keep libtrace well looked-after, even as my available time gets more and more restricted.

15

Mar

2017

Found a number of places in the Chrome source that I can inject headers before they are used for anything, which will allow me to set the Timing-Allow-Origin header for all sites and therefore get proper timing information for objects fetched from any domain. I now need to find a way to set this without having to modify the source - I should be able to replace some part of the stack with my own class that modifies the headers as they go by.

Spent some more time working on the BGP SDN project - added support for the MRT format when reading routing changes, and will currently parse and act on BGP UPDATE messages contained within. Also tried to extract most of the route selection process out into a separate piece of code so that it can be easily replaced/modified. Currently have an implementation of a very basic BGP decision process, as well as a simple decision process of my own that takes into account the internal topology when selecting routes to give to nodes.

15

Mar

2017

Spent some time investigating running the Chrome browser in a headless fashion, and writing code to control it so that it can be used to perform test measurements. The headless mode in development works well, and I can build a test program that will load and control the browser. Using the Chrome devtools API I can evaluate arbitrary javascript on the page, so I have access to the performance timings (but need to deal with the issue of the Timing-Allow-Origin not being set by most sites). My old YouTube javascript timing code still works, but I need to detect the end of the video in order to collect the final stats (Chrome doesn't appear to want to let me hook into arbitrary events, only a disappointingly small subset are exposed).

Worked on the BGP SDN proof of concept some more, making it event driven so that routing and topology updates can be read while it runs - it can now respond to changes in received routes or link status so that the routing tables are always up to date.

13

Mar

2017

Went back and finished making libflowmanager work with parallel libtrace. The remaining problem had been that the expiry modules were not thread-safe, so I've rewritten them to be classes so that the expiry lists are local to each module. Testing with lpi_protoident has proven these changes to work (at least when reading from a trace file), so I can continue updating the rest of the libprotoident tools to be parallel-libtrace compatible soon.

Spent the remainder of my week validating some of the FSMs produced by my model generation algorithm. Overall, the results are starting to look fairly good -- most of the machines being generated by my code are close matches to the ground truth machines, and there are very few duplicate or redundant machines. The most obvious outstanding problem is related to "tandem repeats", i.e. sequences of multiple system calls that can be repeated any number of times (such as "read,write,read,write,read,write", where "read,write" simply repeats until the action is over. Started looking into methods where I could detect tandem repeats so that I can try to encode them as a single self-repeating state.