Week description

The first EvAAL competition was held in the Ciami Living lab in Valencia. During the 3 days of competition, for each competitor the Evaluation Committee has defined and followed a protocol, in order to evaluate the proposed localization systems.

The protocol consists on three phases; namely installation, evaluation, and result.

Installation: When the competitor is arrived the chair has explained the steps of the competition. Later, the competitor has been invited to install its own localization system into the Living Lab. The floor of the Living Lab was completely covered by a carpet that hides to the competitors, the paths that the actor will produce during the evaluation phase. The position of each device deployed on the test site, has been measured and recorded for the metadata purpose. The installation time has been measured in order to evaluate the installation complexity score. This phase ends integrating the competitor's software with the evaluation tool, and unfolding the carpet that hided the paths. 

Evaluation: During this phase only the actor was inside the Living Lab. The score for measurable criteria for each competing artefact has been evaluated by means of benchmark tests. The benchmark was consisted of a set of tests, each of which has contributed to assessment of the scores for the artefact.

Result: The last phase of the competition was related to the interview of the competitors, regarding integrability aspects. Later, each member of the evaluation committee gave its personal score about the user acceptance that, together with the previously evaluated scores, has composed the overall score. The competitors have filled a questionnaire about the organization of the EvAAL competition, and they have gave to the organizer the intermediate data produced during the evaluation phase in order to collect the datasets.