NOVA is a tool for annotating and analyzing behaviours in social interactions. It allows to visualize data recorded with the SSI Framework, as well as from external sources. A main feature of NOVA is that it allows to employ a collaborative annotation database where annotation work can be split between multiple sides, but also between a human annotator and a machine by supporting human annotators with machine learning techniques already during the annotation process - A process we call Collaborative Machine Learning.

NOVA allows framewise labeling for a precise coding experience, and value-continuous annotations for labeling e.g emotions or social attitudes. The interface is customizable and allows loading and labeling data of multiple persons.

The Annotation format can easily be imported in other tools, like ELAN or Excel. NOVA further supports the Import of Discrete Annotation files from ELAN and ANVIL for a seamless workflow.

The Cooperative Machine Learning capabilities allow to train and evaluate machine learning models, such as Support Vector machines or Artificial neural networks directly from the interface with both, a session completion step, where a model is trained on the first minutes of an annotated sessions to predict the remaining session, and a session transfer step where a model is trained on multiple sessions to predict completly unknown data. With the help of human input the models can be refined. The collaborative ML workflow is illustrated below.

The latest binaries can always be found here.

FAQ:

Help, I opened a video but it doesn't show up

Please make sure you installed the according Video Codec on your System. E.g. Shark007 Codecs might be a good solution for most missing codecs, which enable you to control audio/video codecs with dedicated GUI. Also make sure you installed a Visual Studio 2015 redistribution package if it doesn't come with your system.

I don't know what to do, where can I get help?

The offical documentation can be found here. If you run into trouble, please create an issue on git.

Will NOVA run on my Mac/Linux Machine?

As NOVA is an WPF Application it will run on Windows. Nevertheless you can of course use a Virtual Machine to use it.

I found a bug, can you fix it?

NOVA is Software under development and is provided “as is”. If you run into any problems or find bugs (or want to contribute to the project) feel free to open an issue here on github.

Is there an example pipeline to automatically create annotations?

check out https://github.com/hcmlab/kinect2-record for a kinect 2 example recording pipeline.

I would like to contribute to the project

Please feel free to fork or create an issue

Publications:

If you are using NOVA in your research please consider giving us a citation:

"eXplainable Cooperative Machine Learning with NOVA", Tobias Baur, Alexander Heimerl, Florian Lingenfelser, Johannes Wagner, Michel F. Valstar, Björn Schuller, Elisabeth André, in Ki - Künstliche Intelligenz, 2020

@Article{Baur2020,
author="Baur, Tobias
and Heimerl, Alexander
and Lingenfelser, Florian
and Wagner, Johannes
and Valstar, Michel F.
and Schuller, Bj{\"o}rn
and Andr{\'e}, Elisabeth",
title="eXplainable Cooperative Machine Learning with NOVA",
journal="KI - K{\"u}nstliche Intelligenz",
year="2020",
month="Jan",
day="19",
issn="1610-1987",
doi="10.1007/s13218-020-00632-3",
url="https://doi.org/10.1007/s13218-020-00632-3"
}

“NOVA - A tool for eXplainable Cooperative Machine Learning”, Alexander Heimerl, Tobias Baur, Florian Lingenfelser, Johannes Wagner, Elisabeth André, in Proceedings of the 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), Cambridge, September 2019

@INPROCEEDINGS{Heimerl2019,
author="Heimerl, Alexander
and Baur, Tobias
and Lingenfelser, Florian
and Wagner, Johannes
and Andr{\'e}, Elisabeth",
booktitle={2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)},
title={NOVA - A tool for eXplainable Cooperative Machine Learning},
year={2019},
pages={109-115},
doi={10.1109/ACII.2019.8925519},
ISSN={2156-8103},
month={Sep.}
}