User login

Shane Alcock's Blog




Libtrace 4.0.0 is now out of beta and considered ready for general release.

We've fixed quite a few bugs over the course of the beta. More details can be found on the ChangeLog page on libtrace wiki. However, while we're no longer in beta, there may still be a few bugs out there -- don't hesitate to report any problems you find to us at contact [at] wand [dot] net [dot] nz.

Another major change since the beta release is that we've re-licensed libtrace and libpacketdump to be under the LGPL v3 (rather than the GPL v2). Hopefully this will encourage people who were turned off by the restrictions of the GPL to now adopt libtrace for their packet capture and analysis needs.

This version of libtrace includes an all new API that resulted from Richard Sanger's Parallel Libtrace project, which aimed to add the ability to read and process packets in parallel to libtrace. Libtrace can now also better leverage any native parallelism in the packet source, e.g. multiple streams on DAG, DPDK pipelines or packet fanout on Linux interfaces.

Please note that the old libtrace 3 API is still entirely intact and will continue to be supported and maintained throughout the lifetime of libtrace 4. All of your old libtrace 3 programs should still build and run happily against libtrace 4; please let us know if this turns out to not be the case so we can fix it!

Learn about the new API and how parallel libtrace works by reading the Parallel Libtrace HOWTO.

Download the new release from the libtrace website.




Libwandio 1.0.4 has been released today.

The main change in this release is that the licensing has moved from GPL v2 to LGPL v3.

The other major change is that we've hopefully finally fixed all of the segmentation faults that would occur if you used wandio on a 32-bit system.

More details on the changes in this release can be found in the Changelog file included with the libwandio source code.

You can download the new version of libwandio from our website.




Released new versions of libprotoident and libflowmanager with the new LGPL licensing. Also re-licensed and tested potential libtrace and wandio releases but haven't quite got to the stage where I want to push out the releases just yet.

Continued messing around with deriving FSMs from common system call patterns and turning them into runnable code. I've got 8 FSMs drawn up and have implemented 5 of them. Developed a bit of backend for applying my FSMs to the log data so that I can implement new FSMs with the least amount of coding possible (e.g. common actions like checking fd consistency and making sure paramaters match expected values are all done within a parent FSM class and the child classes just list the relevant data to compare against). Hopefully this will help move towards automated generation of the FSM code.

Had a few meetings where we discussed the FSM approach (and RA3 in general) with a few of the industry partners and they seem reasonably pleased with what we are trying to achieve so that's reassuring.

Helped Brendon try to debug some issues with data not appearing on graphs on the recently updated deployment. As a result of this, we've realised we need to re-think how we are storing and presenting traceroute data so that we can't avoid these problems in the future.




A new version of libflowmanager has also been released today.

Once again, the main change is that the licensing has moved from GPL v2 to LGPL v3.

We've also made some changes to make it easier to experiment with different flow expiry algorithms. Flow expiry behaviour is now implemented as separate plugins, rather than being hard-coded into libflowmanager itself. This means if you like the structure of libflowmanager but don't agree with our timeouts for inactive flows, you are able to write your own without having to touch the core of the library. We also added a couple of other config options that allow you to further tweak timeout behaviour -- see the ChangeLog included with the source code for more details.

You can download the new version of libflowmanager from our website.

We've also put libflowmanager up on our github, so you can follow any future libflowmanager development more closely.




Libprotoident 2.0.9 has been released today.

The biggest change in this release is that libprotoident is now using the LGPL v3 license rather than the GPL v2 license. We hope that this will be welcome news to some people who had previously wanted to use libprotoident in their software but were put off by the restrictions of the GPL license. Note that we are aware that our other libraries (libtrace, libflowmanager, wandio) that libprotoident depends on are still GPL -- rest assured that LGPL versions of these libraries will appear soon.

We've also added support for another 12 new application protocols, including Facebook Messenger, Facebook Zero, Overwatch and Baidu Yun P2P. We've improved the rules for a further 16 protocols such as Google Hangouts, Minecraft, QUIC, World of Warcraft and DOTA2.

As always, the full list of changes can be found in the libprotoident ChangeLog.

Download libprotoident 2.0.9 here!




Started looking at the most common patterns in my example sysdig logs. It's pretty obvious that we can easily recognise some low-level actions based on the sequence of system calls and produce models that can be used to identify them. For example, loading a .so shared library will generally result in the same sequence of system calls (with some minor variations) and therefore that can be expressed as a finite state machine.

Developed FSMs for four low level actions: loading a .so library, loading a python module, receiving a typed character via ssh and reading a modprobe config file. Implemented the SSH action as code so I can now find and replace those sequences in the logs with a single SSHCharInput action.

Helped Brendon install NNTSC, ampy and amp-web packages on one of our existing deployments on Thursday. We ended up with a problem where NNTSC would not return query data to the web-site and it took a lot of time (and debugging) to find the source of our problem: incongruous versions of psycopg2 in pip vs the debian package.

Started prepping a libprotoident release. libprotoident is moving to an LGPL license so I've had to replace the blurb at the top of every source file. Been working through the usual pre-release testing and ChangeLog updating.

Spent Wednesday at the Honours conference. I thought all of our students presented well and gave good accounts of their work so far.




Worked on the camera-ready version of my IMC paper. Managed to add some nice content to address the reviewer feedback we got, only to find that I had been using a font size that was too small (i.e. the default font size that every previous IMC has ever used). Unfortunately, switching to the bigger font size would mean I would have to remove almost all the new content I had added, so I'm hoping the PC chairs will change their mind and revert back to the old font size.

Wrote the basic architecture for a provenance log parsing library that can be used with both live progger records and sysdig log files. This will replace the old progger-central which I had written as a hacky PoC which was in danger of becoming production code otherwise.

Got my script to extract common patterns from Sysdig logs working reasonably well. Took a few attempts to get some nicely formatted output that contains all the information I should need to track down what actions are causing the repeated patterns.

Spent a fair bit of time helping CROW get a handle on the Endace Probe, what it can do and how it might fit into their research goals.

Listened to our 520 student practice talks on Thursday. The projects themselves are pretty good -- just the usual issue of the students underselling just how much actual work they had put in to the development side of their project.




A short paper by myself, JP Möller and Richard Nelson titled "Sneaking Past the Firewall: Quantifying the Unexpected Traffic on Major TCP and UDP Ports" has been accepted for publication at this year's upcoming IMC. We'll post the final version of the paper once I've finished making the final revisions, but feel free to get in touch if you want a sneak peek.

As part of this research, we spent a lot of time investigating traffic on TCP and UDP ports 53, 80, 443, 8080 and 8000 that did not match the 'expected' application protocol for that port. At the outset of this work, the vast majority of the traffic was unable to be identified by libprotoident so we ended up adding or improving quite a few libprotoident rules. Our reviewers were particularly interested in the new rules that we created but space limitations in the paper itself mean that we are unable to include a lot of detail about the new rule developments in the text.

Therefore, this page is intended to serve as an addendum to the published paper by explicitly stating which protocols were identified as a result of the research paper and provide links to the source code in libprotoident where the new rules are defined.

Entirely New Applications (21)

  • the purpose of this protocol is not entirely clear but the remote hosts involved are typically owned by (a Chinese antivirus company).

  • 360 Safeguard: update protocol used by 360 Safeguard, a Chinese antivirus.

  • Airdroid: Application for remotely controlling Android devices from a desktop computer.

  • Bad Baidu: Strange behaviour observed on hosts with the Baidu web browser installed. Appears to be some sort of phone-home protocol, but manages to blatantly violate TCP specs in the process.

  • Dianping: Chinese online-shopping and establishment rating smartphone app. Also has a UDP protocol.

  • Kakao: Korean messaging and chat for smartphones.

  • Kankan: Chinese Video streaming service. Also has a UDP protocol.

  • Kuaibo: Chinese Video streaming service.

  • Kugou: Chinese Music streaming service.

  • Norton Backup: Backup and recovery service run by Norton, better known for their antivirus products.

  • QQ Download: File downloading software created by Tencent, who are also behind the popular Chinese messaging tool, QQ.

  • QQ PC Manager: Anti-malware software created by Tencent.

  • Telegram: Cloud-based messaging service with an emphasis on security.

  • Tensafe: Anti-cheating software that is integrated with major online games published by Tencent in China (such as Blade and Soul).

  • Weibo: Chinese microblogging service.

  • Wolfenstein: Enemy Territory: Free online multiplayer game, released in 2003 but still played.

  • Xiami: Chinese Music streaming service, owned by Alibaba.

  • Xunlei JSQ: Game acceleration service from the company behind Xunlei (a.k.a. Thunder).

  • Xunlei VIP: Fast download service for VIP users of Xunlei (Thunder), which pulls cached content from Xunlei servers rather than the standard P2P from other Xunlei users.

  • Xunyou: Chinese game acceleration service.

Existing Protocols Improved (10)

  • DNS: Protocol for mapping hostnames to IP addresses. If you're reading this, you should know what DNS is for.

  • Fortinet: Protocol for updating Fortinet network appliances.

  • Kaspersky: Russian security software.

  • NTP: Time synchronisation protocol.

  • QQ: Very popular Chinese instant messaging application.

  • QUIC: Protocol originally developed by Google for transferring streamed content (especially YouTube video) over UDP.

  • Taobao: Chinese online marketplace, similar to Amazon.

  • WeChat: Another popular Chinese messaging application.

  • Xunlei: Also known as Thunder. A Chinese file sharing system which also leverages other P2P technologies, e.g. BitTorrent, eDonkey etc.

  • Youku: Chinese video hosting / streaming service, somewhat analogous to YouTube.




Got NNTSC and amp-web working with the sysdig data that Harris gave me, so we have a simple proof-of-concept. After talking with Harris some more, he is interested in finding patterns in the syscall logs that are "predictable" so that we can build models of known specific actions on a system (e.g. opening a file with vim, starting a python interpreter etc). Started working on a script to identify common patterns in the sysdig logs so that we can get an idea as to what these patterns look like and how hard they will be to recognise and identify.

Continued tracking down unknown traffic patterns with libprotoident. Managed to nail one pattern that had been bugging me for a long time: the Baidu Yun P2P protocol. Also added rules for YY, Overwatch, Zoom TCP and NetCat CCTV.




My IMC paper on unexpected traffic on well-known ports was accepted, which is great news. Spent Monday going over reviewer feedback and thinking about what revisions I need to make for the camera-ready version.

Continued working on integrating STRATUS with NNTSC. Spent way too much time trying to figure out why my data was not being inserted into the Influx database -- turns out the timestamp for the test data I was using was too old for the default retention policies so it was being automatically discarded. Fudged the test data times to be more recent and it finally worked.

Added file operations metric support to ampy and amp-web so we can now look at simple graphs of open frequency data. Found some scalability issues with our modal dialogs in cases where the number of possible options for a dropdown is very high, so I've gone back and added pagination support to all modal dropdowns so they only load 30 or so options at a time. This had some interesting flow-on effects, especially for the latency modal dialog which had a lot of custom code for populating the tabs for the different latency metrics. I think I've ironed out all of the extra wrinkles now.

Spent a little more time with the July traces to track down some more unknown protocols. Added a rule for the Netcore vulnerability scan (which happens a lot!) and updated rules for a lot of (mostly game-related) protocols.