User login

Blogs

20

Nov

2017

Implemented a few more nDAG optimisations. I'm now using recvmmsg and sendmmsg to receive and send batches of messages using a single system call -- this saves us quite a lot of pthread cancellation status changes when doing I/O. Sender can now comfortably handle 7.5 Gbps without dropping any records. Client is still stuck around 5 Gbps receiving, but I still have one or two ideas up my sleeve.

Tested and released a new version of libtrace, which includes both nDAG read and pcap-ng read support.

Started experimenting some more with the ETSI capture format. Started adding an etsilive: input format to libtrace for testing purposes. Played with cyberprobe as a possible independent source of ETSI-encoded packets to develop my decoder against, but it turned out that their encoding was not as compliant as I had hoped. I found two errors in their encoding of the first field alone.

15

Nov

2017

Libtrace 4.0.2 has been released today.

This release adds two new input formats: pcapng and nDAG.

This release also includes the following bug fixes:
* Fixed bad IPv6 fragment offset calculation that would randomly cause IPv6 packets to be incorrectly parsed.
* Fixed bug where SIGINT would cause parallel programs to assert fail.
* Fixed compilation issues caused by inconsistent BPF presence checks.
* Fixed errors in IPv6 and SCTP libpacketdump decoders.
* Fixed bug where a short cryptopan key would cause traceanon to assert fail.
* Fixed compilation error when building traceanon against new versions of libssl.
* Fixed bug where DPDK would not be detected if it had been built as a shared library.
* Fixed bug where the wrong parallel read function would be used by libtrace.

The full list of changes in this release can be found in the libtrace ChangeLog.

You can download the new version of libtrace from the libtrace website.

13

Nov

2017

Continued working on tweaking nDAG to both improve performance and add some handy features such as the ability for clients to recognise when an nDAG monitor has restarted and therefore may have missed some packets. Still got one or two ideas on how to improve performance further, so will try those out before merging the code back into mainline libtrace.

Started thinking a bit more about how my ETSI monitor is going to work and how much of it will intersect with libtrace. Will probably need to add an etsilive: read format to libtrace with suitable libpacketdump decoders to help with testing and validation, so that seems like a useful starting point.

Added a feature to my daily libprotoident analysis program to tell me what proportion of traffic on the campus network remains unidentified.

09

Nov

2017

Spent some time working on the packaging scripts to upgrade ampy/ampweb/netevmon to the newest version, including moving some database tables around, populating new tables and dealing with the debconf answers during install. Installed these multiple times on a few different Debian flavours while trying to make sure that they all work.

Found and fixed a few issues in the ampweb matrix that were preventing the udpstream data from displaying properly. There are still a few issues here around udpstream data being used to generate latency graphs, but it's getting closer to working in every case.

Continued to work on tidying up some of the BGP router code.

09

Nov

2017

Found and fixed a bug where the udpstream test would not receive all the reflected packets used for RTT calculations because it gave up listening after waiting after one inter-packet gap. It now waits until all the packets have been received or the global loss timeout is reached (multiple seconds). Also found and fixed the problem that led to discovering this - the web interface was asking for inter-packet gap in milliseconds and then treating the number entered as if it was in microseconds, leading to a gap 1000 times smaller than expected.

Added the ability for an ampweb user to change their own details through the web interface without requiring admin privileges. Spent some time testing that the new permissions model works correctly and that users are limited appropriately.

Started work on adding debconf support to the ampy package as a simple way to ensure there is a usable user right from the start without hardcoding one.

07

Nov

2017

Last week I kept on reading through more traffic engineering papers. The papers that I am currently looking through on TE cover different network types as well as look at various TE goals, such as resource utilisation optimisation, QoE maximisation and congestion minimisation. They were found from a literature review paper that looks at how SDN can benefit TE. The papers that I am focusing on look at OpenFlow and TE. Last week I have also finished and ran my Fast-Failover group timing tests.

This week I am planning on finishing off reading the TE with OpenFlow papers from the literature review. I also want to try and run more tests and see if I can get a sense of how long a recovery based error detection method (not using fast failover groups and precomputation) takes to complete. I currently have results for protection and would like to compare them with recovery. I am also currently in the process of looking for source code for some of the error detection and recovery systems presented in the papers I have read through. I would like to run some of these systems or methods and potentially assess their behaviour, problems and performance.

06

Nov

2017

Finished adding the core of nDAG client support to libtrace. Still a little bit of polish required before it is officially finished, but it seems to work. Managed to get around 3.5 - 4 Gbps of multicast to the libtrace client without losing anything, which is not too bad. Once I increase the data rate, it looks like the switch is dropping multicast packets rather than the client themselves so I may be starting to run into some hardware limitations.

Spent a bit of time playing around with libtasn1 and the ETSI ASN.1 specification to see how I can use the library to create some ETSI headers for packet encapsulation. Went public with a proposal for an open-source ETSI lawful intercept tool on Friday and have already got some encouraging responses.

Still seeing new patterns in the Waikato traffic, so libprotoident continues to improve. Reached 450 supported protocols this week -- next landmark is 500.

31

Oct

2017

Last week I carried on reading through a few more traffic engineering papers and have also looked at a couple of SDN controller performance evaluation papers. At the end of the week, I started to set-up VM environments to run some quick tests and hopefully benchmark some of the solutions available for error detection.

This week I have a few more TE papers to go through. I also want to run some tests to find out how long the fast failover group takes to switch to a new bucket when a link goes down. I also want to try and run a few benchmarks on some the current error detection and recovery methods/systems from the papers I have read.

30

Oct

2017

Managed to get the new telescope software running at a decent packet rate. So far we can capture and multicast ~12 million packets per second without issues. The main limitation that prevents us from going any higher is the capacity of the 10Gb interface that we are multicasting on. Pretty happy with that result and now I can focus on ensuring that the clients will be able to keep up.

Started adding nDAG read support to libtrace. This is mostly a matter of adapting my existing test client code to work within the libtrace structure, as well as making sure that there are suitable code paths for each of the three APIs: parallel, single-threaded and event-driven.

Still seeing new protocols every week on the campus network, even with the decreasing amount of people who are present on campus. 3 new protocols this week; starting to get close to the 450 mark.

24

Oct

2017

Updated the AMP user UI to allow users to view/modify their own details, which required changing the way permissions were tested in a few places to properly control access. Tidied up some of the modal dialogs to properly update the different parts of the form in response to user input, hopefully making it easy to see what needs to be fixed/completed before the form can be submitted.

Spent some time tidying up the source for the BGP router and trying to make sure that the style is consistent across all the source files. Also ran some static analysis/lint tools over the source to help make sure we are being sensible.