User login

Blogs

14

Sep

2016

Finished up the merging algorithm for now. Most cases are handled correctly, including more complex cases such as when the flow rule is modified and then matched on the new value in a latter table. The only case that will need better handling is the case that a single OpenFlow rule cannot be used to represent the multiple rules. For instance this occurs when matching QinQ, only one vlan can be matched per table making it impossible to match both QinQ tags in a single table.

I've started looking at getting my recent AT results into the packet in and out paper and looking for a conference to submit to. SOSR seems to be a good choice and gives me just over a month to get it ready. So I will be putting my main focus back into that. I also have my 6 monthly PhD report due this month so I'll be getting that together also.

13

Sep

2016

Last week, I added a configuration to RheaFlow's configuration file to make TTL decrements on the switches optional because the some of the tests conducted showed that some OpenFlow switches are not able to perform this action in hardware. I also conducted more tests with the Pica and ZodiacFX switches, both switches are unable to process IPv6 related flow rules because in the case of the ZodiacFX switch IPv6 is not fully supported while the Pica requires multi-table rules to process IPv6 related flows because the IPv6 fields are too large to be processed by the switch hardware in a single table. I also tested multiple switch configuration with RheaFlow in slow path mode with two switches.

I'll be working on the code to handle packet forwarding to the controller via inter-switch link this week and verifying different configuration mode with multiple switches. I'll also get started with writing the paper for the RheaFlow.

12

Sep

2016

Fixed up another couple of minor issues that had been reported. Fixed the loss timer in the tcpping test to start after the last packet is sent so that long interpacket delays are possible if desired. Tidied up a regex to match certificate filenames more accurately. Made more documentation updates. Tried to improve packaging to make sure that default configuration files were as usable as possible without manual edits.

Built new packages and pushed them out to one of our test deployments. Worked through a few issues with getting the HTTP tests and target meshes lined up properly so that they display. Still need to figure out the correct way to fix this so that users don't need to worry about this special case.

Spent some time reworking my Debian package build system after accidentally building with incorrect source due to some release candidate versioning suffixes being missed. The new system will also better deal with release specific Debian directories.

12

Sep

2016

Finished up the libtrace4 and wandio releases and pushed them out.

Installed a mock version of skeptic on an openstack VM to test how InfluxDB copes with the full public AMP dataset. In general, InfluxDB seems to be coping OK when inserting / browsing data but the memory requirements of anomaly_ts are a bit larger than I would like so that's an avenue to chase up in the near future.

Continued implementing syscall FSMs manually to find out about other cases we need to consider when trying to automate the process. Added the ability to express a state as another FSM so we can build more complex machines from the smaller ones. Documented the code and put it into bitbucket so other people can start working with it.

Also started trying to use the FSMs on another dataset that Alan had collected. Turns out this dataset had a bunch of new syscalls that my previous parser hadn't seen before so it required a bit of updating.

07

Sep

2016

I updated RheaFlow's to code to make all compoenents start together using systemd. I also tested RheaFlow with Allied Telesis AT-x930 switch. While testing RheaFlow on the Allied Telesis switch, it was discovered that the switch does not perform TTL decrements of IP packets in hardware which impacts forwarding performance, I've started updating the RheaFlow code to include an optional configuration to order OpenFlow switches managed by RheaFlow to decrement TTLs.

I fixed RheaFlowProcessor.py's handle_packet_in method to log and discard packets received from the virtual switch's (dp0) OpenFlow LOCAL port. This was considered in the early stages because dp0's instances in my dev environment did not send packets from their OpenFlow LOCAL ports.

This week, I'll complete the updates for the optional configuration for TTL decrements and conduct further tests with the Pica and ZodiacFX OpenFlow switches. I'll also start work on completing the sections of the RheaFlow code that handles inter-switch link configuration and forwarding.

06

Sep

2016

Spent the Monday and part of Tuesday this week getting slides together for the COMP514 lecture I gave on Tuesday. This went well, and I'll be able to reuse some slides in the internal PhD conference.

I then looked at the problem of checking for dependencies beyond pairs - such as triples etc until an entire dependency chain is considered. One realisation I came to was that my simulation when expanded beyond two tables would consider all combinations of the rules - with some situations possibly being detected as unreachable. Something which seems very similar to merging rules into a single table, and just as bad in terms of rule expansion issues. Merging rules into a single table is a procedure I will need in the future anyway to manipulate rules to fit. I've approached this by merging one table with another allowing me to check portions, and the entire pipeline simply by merging the result of the tables prior with the next in the pipeline. This work is still ongoing.

As part of this I have also been adding many set features (intersect, union etc) and other useful operations to the FlowMatch and FlowActions classes. I've also added some more unit testing for adding ActionSets and/or ActionLists together, after finding and fixing a couple of bugs in this.

05

Sep

2016

Made lots of small changes based on things that had been reported by users or that I had noticed behaving incorrectly in the last week. Fixed a cap on a retry timer that was alternating between two different values. Updated the HTTP test to always store the full URL including scheme, even if the user didn't explicitly specify it. Updated some error messages to try to be more useful and accurate.

Fixed the apache2 configuration in the amppki packages to work properly once everything is installed properly in the correct system locations. There were issues around the python path being incorrect and not able to find the libraries, as well as naming collisions with the ampweb WSGI processes.

Spent some time with Shane trying to track down the cause of some missing data in the web graphs. Found the cause of the missing DNS data (wrong column names being used) and why some sites didn't have path length data available (it's only sourced from one style of traceroute test).

Tried to expose through the web interface the ability to force the address family to use when resolving test targets. This was a bit more complicated than expected, due to new targets getting automatically added to the database and it including the various suffixes used internally to represent address families.

Put together some new server packages to test the new changes and started working through verifying that they worked, ahead of another release.

05

Sep

2016

Libtrace 4.0.0 is now out of beta and considered ready for general release.

We've fixed quite a few bugs over the course of the beta. More details can be found on the ChangeLog page on libtrace wiki. However, while we're no longer in beta, there may still be a few bugs out there -- don't hesitate to report any problems you find to us at contact [at] wand [dot] net [dot] nz.

Another major change since the beta release is that we've re-licensed libtrace and libpacketdump to be under the LGPL v3 (rather than the GPL v2). Hopefully this will encourage people who were turned off by the restrictions of the GPL to now adopt libtrace for their packet capture and analysis needs.

This version of libtrace includes an all new API that resulted from Richard Sanger's Parallel Libtrace project, which aimed to add the ability to read and process packets in parallel to libtrace. Libtrace can now also better leverage any native parallelism in the packet source, e.g. multiple streams on DAG, DPDK pipelines or packet fanout on Linux interfaces.

Please note that the old libtrace 3 API is still entirely intact and will continue to be supported and maintained throughout the lifetime of libtrace 4. All of your old libtrace 3 programs should still build and run happily against libtrace 4; please let us know if this turns out to not be the case so we can fix it!

Learn about the new API and how parallel libtrace works by reading the Parallel Libtrace HOWTO.

Download the new release from the libtrace website.

05

Sep

2016

Libwandio 1.0.4 has been released today.

The main change in this release is that the licensing has moved from GPL v2 to LGPL v3.

The other major change is that we've hopefully finally fixed all of the segmentation faults that would occur if you used wandio on a 32-bit system.

More details on the changes in this release can be found in the Changelog file included with the libwandio source code.

You can download the new version of libwandio from our website.

02

Sep

2016

Released new versions of libprotoident and libflowmanager with the new LGPL licensing. Also re-licensed and tested potential libtrace and wandio releases but haven't quite got to the stage where I want to push out the releases just yet.

Continued messing around with deriving FSMs from common system call patterns and turning them into runnable code. I've got 8 FSMs drawn up and have implemented 5 of them. Developed a bit of backend for applying my FSMs to the log data so that I can implement new FSMs with the least amount of coding possible (e.g. common actions like checking fd consistency and making sure paramaters match expected values are all done within a parent FSM class and the child classes just list the relevant data to compare against). Hopefully this will help move towards automated generation of the FSM code.

Had a few meetings where we discussed the FSM approach (and RA3 in general) with a few of the industry partners and they seem reasonably pleased with what we are trying to achieve so that's reassuring.

Helped Brendon try to debug some issues with data not appearing on graphs on the recently updated deployment. As a result of this, we've realised we need to re-think how we are storing and presenting traceroute data so that we can't avoid these problems in the future.