User login

Blogs

28

Nov

2016

Back to work on Wednesday after my brief holiday in LA. Caught up on a backlog of email and bug reports, then started getting back into my FSM generation work for STRATUS.

Fixed some bugs where my code was incorrectly introducing loops into the auto-generated FSM. Continued testing with some of the logs that Alan had collected from specific applications and making sure that the resulting machines appear sane.

The most obvious outstanding problem now is that my original pattern extraction approach does not recognise cases where a pattern consists of a much shorter subpattern that is being repeated. I'd rather be using the subpattern to create my FSMs, otherwise I end up with FSMs that have a path for pattern X being repeated 4 times, X being repeated 5 times, etc. based on what combinations I saw in my original log. After drawing a lot of suffix trees on the whiteboard, I think I've come up with solution that might work -- next week's job is to implement and evaluate it.

23

Nov

2016

Updated many of the pyramid routes to more accurately reflect the URLs that they are expecting and to make sure that only correct URLs containing all required parameters are passed to each route. Also fixed up many instances of generating route URLs used in href targets to properly set all the attributes.

Continued tidying up the new REST interface. Updated various scheduling/site/mesh queries to be better behaved when presented with odd data (distinguish between errors and lack of data), idempotent where possible. Added the ability to hide meshes from the matrix, so that various internal meshes used for scheduling etc don't show up, only those that you want to expose. Started writing some basic documentation about what endpoints are available and what they do.

Spent some time writing up ideas for a possible NZNOG AMP tutorial.

23

Nov

2016

Double report due to being away at IMC.

Gave a practice run of my IMC talk. Still needed to carve out some content and streamline it a bit, so spent more time working on that.

Documented the process for adding a new metric (i.e. a new graph/matrix for an existing collection) to the AMP website. Worked through an example by adding HTTP page size to my development website. Identified a number of issues with some of the terminology in the amp-web code that need to be fixed in the long run, but this will probably require a decent bit of code re-architecting.

Attended IMC in Santa Monica. Talk seemed to go over pretty well and it was great to catch up with people I had met at AIMS earlier this year, as well as some familiar faces from previous IMCs. Took the opportunity to have a brief holiday in L.A. afterwards.

22

Nov

2016

Continued working on getting the Table Type Pattern Validator together. This is mostly done now, just html cosmetics left now.

I've gone through and removed most of the asserts in the code and either replaced them with error messages or proper exception handling, which reduces the chance of no result being returned. I've added a simple cache so that common patterns won't need to be recomputed. I've added a number of simple tests to check some code paths I fixed. Added proper unicode detection allowing for UTF-8, 16 and 32 as per the JSON spec. Added rate limiting, and fixed an issue with undefined variable names persisting globally.

18

Nov

2016

Began working on my summer research project. Have been familiarizing myself with Python as Matthew recommended completing the project in a scripting language. Brad sent through some links to Python resources. I have read majority of the 'Dive Into Python 3' book and the (official) Python 3 Tutorial. Have also starting looking at the data sets that I will be using for the project.

18

Nov

2016

Half the week was spent finishing up the packaging of the server components, and then installing them on amp.wand.net.nz. Still had some annoying database permissions issues that show up on fresh installs, but slowly eliminating them as I find them.

Chased down and fixed a few bugs that had been reported, including some unsigned latency values that were wrapping to very large values, and throughput graphs breaking in places where division by zero was possible due to unchecked values.

Spent some time installing and configuring event detection packages on another site. Ran into some issues where events weren't being generated and worked with Shane to try to track it down. The issues seemed to be related to having collections enabled but with no tests being performed - there was no data to fetch so it didn't realise that it was up to date.

Got a small amount of work done tidying up the scheduling web interface to be more easily exposed for others to use. Planned out the URL structure, updated most of the interface to use the new URLs and started to add a few new functions that could be useful.

15

Nov

2016

Worked on getting together a talk for the Internal CS PhD conference. Presented this on Friday, I think it went well. The slides were based on the slides I made for a COMP514 lecture.

I've made initial steps towards getting a TTP validator online. Brad suggested I use Flask, so I've been working on integrating my existing code into Flask. I'm working through the basics of getting a simple web page together and getting input into the program while updating error handling to save errors directly against the TTP-object rather than just logging.

I've still got a number of issues to work through, replacing any asserts left in the TTP code with warning messages. And other abuse considerations such as rate limiting, client validation and correct handling of input errors JSON.

07

Nov

2016

Final week working on the paper, a week of fixing all the mistakes everyone found. I'm hugely thankful to Shane, Matthew, Brad and Richard all taking time to give feedback. I've submitted the paper to SOSR this week.

Next week(s) is getting a presentation together for the CS PhD conference, and working on getting a TTP validator released online and getting back into getting a first basic working algorithm together to translate OpenFlow rules to different pipelines.

07

Nov

2016

Spent the week working on updating packages for all the server-side components (ampweb, amy, nntsc, netevmon, etc). This was made harder than it should have been by me making the build dependencies list more accurate, causing a different build system to be used, which then tries to pull in further dependencies from PyPI (that are unavailable as packages in some Debian flavours) and build them on a machine that doesn't have (and shouldn't have) the libraries to do so. Ended up reverting some of the changes in the interests of getting it to work, and will revisit this another time.

Made various improvements to the packaging scripts, such as tidying up enabling events when ampweb and netevmon are both present, regardless of the order that they were installed, installing the new influxdb database, fixing permissions, and making sure users exist. Went through multiple test deployments on both Wheezy and Jessie to make sure that the upgrade process will work smoothly next week.

07

Nov

2016

Continued reading over versions of Dimeji's and Richard's papers and providing them with (hopefully) useful feedback.

Continued working on my IMC talk. All of the content is in place now, but it's probably a bit long. Will try to refine more before I leave.

More tweaking of the syscall analysis code to try and make sure the output matches my expectations in each example that I've tested it against. The algorithm still doesn't work too well in the presence of multi-syscall loops, so I'll need to think of an approach to recognise and represent loops rather than treating each iteration of the loop as a branch.