User login

Meenakshee Mungro's blog

05

Aug

2013

Started the week by doing some reading and going over the theory behind a forecasting technique called Adaptive Response Rate Single Exponential smoothing. I then spent the next few days implementing and testing a detector that uses the smoothing technique to obtain the next forecasted byte count. Still need to figure out how to tweak the parameters so as to obtain a detector that does not produce a delayed copy of the actual measurements.

Plan for next week is to do some reading on event detection in time series data, especially looking at techniques/methods that NetEvMon does not currently use.

29

Jul

2013

Spent the week taking a look at the detectors written by Shane and Brendon and managed to cover them all and have a good understanding of how they work. Also had a brief look at the theory behind the Dempster-Shafer belief functions, but need to go through them in more depth at some point.

Plan for next week: start reading up on event detection in time-series data and get set up with my own copy of netevmon. Also look into other smoothing functions that could potentially be useful for new detectors.

23

Jul

2013

I started working on my Masters project on the 8th July. My project is titled "Rating the Significance of Detected Network Events" and so far, I have read parts of Andreas' thesis to get a better understanding about anomaly detection techniques and belief theory. I have also spent a fair amount of time looking over the detectors being used by Netevmon. Currently, I am in the process of documenting the current state of the event detectors for possible inclusion in the thesis, but mostly for a better understanding of the current system, its shortcomings and future extensions.

The plan for next week is to finish documenting the remaining detectors(3 left), and get some reading done about event detection in time-series data(not necessarily network time-series data) and look at some methods used by Netevmon(Arima-Shrewhart) and Andreas(Dempster-Shafer). I also want to have a better look at a forecasting method Andreas wrote about and implemented: Adaptive Single Exponential Smoothing since I will probably try to test it at some point in the future.

24

Mar

2013

Got a full draft of the report, with a final version of Chapters 2-5. 9 chapters spread over 56pages of (hopefully) dry enough material.

Plan for the next 2 days is to edit, edit and then edit some more.

Glad it'll be over soon.

18

Mar

2013

Spent the week fixing up chapters 2-5 and emailed Richard a copy on Friday night(including chapter 6). Didn't get to work on any new chapters during the weekend, had assignments to catch up on.

Plan for this week is to finish chapter 7 by Wednesday, and start on chapter 8 and 9.

12

Mar

2013

Spent the last week working on the report. Have a draft version of Chapters 2-5 that have been checked atleast once by Shane, and added most of the content for Chapter 6.

The plan for the coming week is to finish chapter 6, get it checked asap and give Richard a copy of chapters 2-6.
Then, I have a decently sized chapter to work on(7 - Threaded Network Export) and 2 smaller ones(8-Testing/Evaluation and 9-Future Work/Conclusion). There's also the intro that I need to edit at some point.

05

Mar

2013

Spent the first half of the week working on the collector. Implemented exporting expired flow records and designed another protocol header and subheader for these records. Cleaned up some repetitive code and added a function to export the ongoing flow buffer when the timer expires(before checking for new ongoing flows). Also added some documentation.

Started working on the report in the middle of the week and so far, have a draft version of the first 4 chapters(excluding the intro). Shane has checked a couple of them already so the plan for the coming week is to tidy up those chapters and get as much writing done as possible.

26

Feb

2013

Shane suggested sending the protocol names once only to reduce the amount of redundant data sent each time and also, save on fifo space and bandwidth requirements. I designed a new protocol subheader for exporting protocol details(id, name, name_len) and these are sent to a client as soon as it connects to the server. Then, I had to chage the old exporting code and get rid of parts adding the name and name length and add in the appropriate code for the protocol IDs.

Then, I started working on exporting expired flow records to clients every X seconds(where X = 3mins/value chosen by user). I created a subheader for expired protocol records, and a structure for an expired flow record. Each time a flow expired, it was sent to be exported and its data added to the appropriate buffer. The buffer was then written to the FIFO when it filled up.

After I made sure that expired flow records were being exported correctly, I setup a timer which would export these records every X seconds, regardless of whether it was full or not.

Also got my Background chapter back from Shane. and started making the proposed changes.

18

Feb

2013

Spent a major part of the week reading up on and adding threading and using libfifo with the collector.

First, I added support for using Libfifo in order to write the buffer to a memory-backed FIFO with a default size of 100MB(which can be changed via options). Then, I wrote out the FIFO to each of the clients through their fd by using the functions provided in the Libfifo API. Tested and got it working like before.

Initially, clients that connected to the server were sent statistics every X seconds(where X is a number specified in the options). Concurrency issues would arise when clients would try connecting/disconnecting during a stat export, which implies that the client list would need to be updated while the exporting process was iterating over it. After discussing this with Shane, we decided to use threading and to create mutexes around the client list when it was being read from/written to. The server can now handle disconnects/new connections while exporting statistics without crashing.

Spent Friday at home and got started on writing the Background and Libprotoident chapters of the report(ch. 2 & 3 respectively). Worked during the weekend too and nearly done with the Background and almost half-way through the 3rd chapter.

Plan for the next week is to get the background section's draft done asap and move it to LaTex and get it checked before the end of the week if possible. I also have a list of features that I need to tackle in the collector.

13

Feb

2013

Spent the first few days of the week working on my presentation and then spent the whole Friday taking care of some tickets.

Previously the server was not handling disconnects from clients, so it would still try to send data to the file descriptors. I fixed that first and then worked on not sending statistics for deprecated(NULL) protocols, which would enable saving on bandwidth and effort.

For the next week, I need to tackle threading. Which I am not looking forward to.