Improving and Measuring Learning at Cyber Defence Exercises
Name
Kaie Maennel
Abstract
Cyber security exercises are believed to be the most effective training for all training audiences from top (military) professional teams to individual students. However, evidence of learning outcomes for those exercises are often anecdotal and not validated. This thesis takes a fresh look at learning in Cyber Defence Exercises (CDXs) and focuses on measuring learning outcomes. As such exercises come in a variety of formats, this thesis focuses on technical CDXs with Red and Blue teaming elements. The review of adult learning theories and current state of learning measurement in CDXs context are presented. The learning measurements are performed at two CDXs: Locked Shields and Crossed Swords. First one is the largest unclassified live-fire CDX in the world with nearly 900 participants (with Blue teams as main training audience). Second one is a small scale exercise designed to train Red teams. Both exercises are organised by the NATO Cooperative Cyber Defence Centre of Excellence (CCD COE). Such top-end CDXs are highly complex, which makes it hard for organisers and participants to handle. Therefore, both learning design and measurement need careful consideration. This work proposes a novel and scalable learning measurement methodology, called the “5-timestamp methodology”. This method aims at accommodating for both—effective feedback (including benchmarking opportunity) and learning measurement. The method is capable of assessing team performance, and argues that changes in performance over time equal learning. The timestamps can either be collected using traditional methods, such as interviews, observations and surveys, but also potentially be obtained non-obtrusively from raw network traces (such as pcaps). The method enhances the feedback loop, allows identifying learning design flaws, and provides solid evidence of learning value for CDXs. Crossed Swords measurement focused on providing the training audience (Red team) with instant feedback about their actions to ensure effective learning. This work contributes to theoretical foundations and in practical terms by providing practical recommendations readily applicable for improvement of learning experience in CDXs.
Graduation Thesis language
English
Graduation Thesis type
Master - Cyber Security
Supervisor(s)
Rain Ottis, Liina Randmann, Raimundas Matulevičius
Defence year
2017