User login

Shane Alcock's Blog

28

Oct

2016

Spent a couple of days reading over Richard S's paper and providing feedback.

Continued keeping an eye on the influx-nntsc test deployment. Pretty happy with it so Brendon and I will start working on packaging everything and rolling it out to skeptic next week.

Started working on an outline for my IMC talk.

Got some initial results back to Harris and Alan for their experiment using my suffix tree code. Had to rewrite a previously recursive algorithm to be iterative to work with some of the larger syscall logs, since Python is hopeless at recursion.

Migrated the iterative version back into my automatic FSM construction code, which I resumed looking at on Friday. Still finding plenty of cases where variant patterns are not being combined into the original FSM correctly, so this has mostly involved a lot of debugging. The code has started to sprawl a bit, so had to take some time to refactor it into a manageable state.

21

Oct

2016

Spent most of my week working on turning system call patterns extracted by my suffix tree into workable FSMs. So far the focus has been on recognising which patterns are variations on previously seen patterns and creating "branches" in my internal representations of those patterns that incorporate the allowed variations (ideally, without creating any invalid transitions). I also have to account for situations where the pattern has been "shifted", so naively looking at the edit distance between two patterns doesn't work too well.

The other challenge that I've run into, especially with shifted variants where the pattern repeats, is trying to determine the correct start state. In some cases, there are other variants to the pattern that indicate where the start could be -- i.e. the first system call that is common to all variants is probably the starting point -- but this extra information will not always be available.

End result: I've got an algorithm that seems to work as expected on the first couple of examples I've looked at. It'll need more testing on a wider variety of cases and there are still some outstanding situations that I know are not dealt with as well as I would like, e.g. loops that contain multiple distinct system calls.

Changed direction a bit to help Harris and Alan with an experiment they are running that tries to map application log items with system call patterns. Again using my suffix tree code, I am pulling out common system call patterns and reporting the pids and start times of all instances where those patterns appear in the system call logs. Alan will then see how well those correlate with the entries in the application logs.

Spent some time reading over Dimeji's paper and offering feedback.

17

Oct

2016

Added a new collection to NNTSC for storing traceroute path lengths. This allows me to store the path lengths in Influx (for fast matrix aggregation), while keeping the full traceroute data in postgres. Updated ampy and amp-web to use the new collection, so we now have better matrix performance for all data types. NNTSC memory usage still seems to be fairly stable, which is good news.

Made a few final tweaks to my NNTSC paper before submitting on Friday.

Started looking into how I can use the common sequences extracted by my suffix tree code to recognise syscall patterns that can be turned into FSMs. The interesting challenge is identifying and combining variants of the same syscall pattern -- this is still a work in progress but early signs are promising (at least for the one example I've got so far!).

10

Oct

2016

Influx on a skeptic workload was still not performing as well as I'd like, so I've decided to take a different approach. I removed all of the pre-aggregation that we were doing for graph drawing as this was requiring too much computing power for little real gain (i.e. most of the aggregated data would never be used). However I have added matrix-specific aggregations (i.e. minutely and hourly aggregations of the data columns used by the matrix) to try and speed up the matrix loading. These aggregations only look at very recent data and are retained for a short period of time so they are much less onerous to perform.

Initial testing seems to suggest that the matrix loads about 3 times faster than the old postgres system and memory usage by Influx seems to be not ridiculous so hopefully this will prove workable in the long term.

Continued working on my suffix-tree for finding repeated sequences in syscall traces. Initial results are promising but there are a couple of outstanding issues to deal with, mostly related to recognising two sequences as being variants or shifts of one another. If I can do that, I might be able to get close to automatically creating a FSM for those common sequences.

03

Oct

2016

Finished the draft of my NNTSC paper. Got some initial feedback from Brendon which I've been able to incorporate into the paper.

Still not entirely happy with Influx-NNTSC and netevmon running on the same machine, as the combined memory usage will push skeptic's current hardware to its limit. Experimented with running netevmon on a separate VM just to make sure that a remote event database does actually work, so we at least have the option of moving netevmon onto its own dedicated machine.

Finished my implementation of the imprecise pattern mining algorithm. Starting working on a more homegrown algorithm for detecting repeated sequences of syscalls within a larger trace, based on existing techniques for using a suffix tree to find repeated substrings within strings.

23

Sep

2016

Returned to my half-written NNTSC paper with an eye towards submitting it to PAM in a few weeks. Paper is now around 75% finished, including a couple of nice diagrams showing the NNTSC architecture and the database schema. Space is starting to get a bit tight, so I'll have to revisit some of my earlier writing and cleanse it of unnecessary waffle.

With the help of an explanation from Harris, I've been able to decipher the temporal property mining algorithms. Managed to implement the simple version this week, which seems to be doing the right thing, and started working on a
more complicated variant that allows for some imperfections in the source data (e.g 9/10 times a close follows an open, but every now again someone forgets to call close before opening something else).

19

Sep

2016

Kept tinkering with my mock skeptic install. I was a little concerned about the memory usage of anomaly_ts so I went back over some previous work I did to work out relative accuracy rates of each detector under a variety of different parameter settings to try and find good settings for each detector that used a minimal amount of stored history.

Spent a bit of time reading over some papers on mining temporal properties from sequences of function calls. The algorithms that these people are using are a bit tricky to decipher -- the explanation is a bit terse and I don't really have the background in the area to fill in the gaps -- so hopefully Harris will be able to get further than I did.

Continued building FSMs for common syscall patterns. Started working with the user study data which is not at all well covered by my existing FSMs. This appears to be mostly because of various Gnome / X processes and widgets that are continuously polling and receiving events. The syscalls generated by these processes drowns out everything else, so it is hard to find the actions that the users actually performed during the study.

Arranged travel and accommodation for my upcoming trip to IMC.

12

Sep

2016

Finished up the libtrace4 and wandio releases and pushed them out.

Installed a mock version of skeptic on an openstack VM to test how InfluxDB copes with the full public AMP dataset. In general, InfluxDB seems to be coping OK when inserting / browsing data but the memory requirements of anomaly_ts are a bit larger than I would like so that's an avenue to chase up in the near future.

Continued implementing syscall FSMs manually to find out about other cases we need to consider when trying to automate the process. Added the ability to express a state as another FSM so we can build more complex machines from the smaller ones. Documented the code and put it into bitbucket so other people can start working with it.

Also started trying to use the FSMs on another dataset that Alan had collected. Turns out this dataset had a bunch of new syscalls that my previous parser hadn't seen before so it required a bit of updating.

05

Sep

2016

Libtrace 4.0.0 is now out of beta and considered ready for general release.

We've fixed quite a few bugs over the course of the beta. More details can be found on the ChangeLog page on libtrace wiki. However, while we're no longer in beta, there may still be a few bugs out there -- don't hesitate to report any problems you find to us at contact [at] wand [dot] net [dot] nz.

Another major change since the beta release is that we've re-licensed libtrace and libpacketdump to be under the LGPL v3 (rather than the GPL v2). Hopefully this will encourage people who were turned off by the restrictions of the GPL to now adopt libtrace for their packet capture and analysis needs.

This version of libtrace includes an all new API that resulted from Richard Sanger's Parallel Libtrace project, which aimed to add the ability to read and process packets in parallel to libtrace. Libtrace can now also better leverage any native parallelism in the packet source, e.g. multiple streams on DAG, DPDK pipelines or packet fanout on Linux interfaces.

Please note that the old libtrace 3 API is still entirely intact and will continue to be supported and maintained throughout the lifetime of libtrace 4. All of your old libtrace 3 programs should still build and run happily against libtrace 4; please let us know if this turns out to not be the case so we can fix it!

Learn about the new API and how parallel libtrace works by reading the Parallel Libtrace HOWTO.

Download the new release from the libtrace website.

05

Sep

2016

Libwandio 1.0.4 has been released today.

The main change in this release is that the licensing has moved from GPL v2 to LGPL v3.

The other major change is that we've hopefully finally fixed all of the segmentation faults that would occur if you used wandio on a 32-bit system.

More details on the changes in this release can be found in the Changelog file included with the libwandio source code.

You can download the new version of libwandio from our website.