The EvAAL Framework was conceived with the purpose of Evaluating AAL systems through competitive benchmarking. The idea is to gather together working systems, both prototypal and mature, and independently compare their performance in one or several specific areas, with the long- term objective of creating a set of evaluation benchmarks for indoor pervasive systems.

Core and extended criteria

EvAAL Framework's core criteria are:

1. Natural movement of an actor: The agent testing a system moves spontaneously with a predefined script, including walking, resting or other actions.
2. Realistic environment: The scene is a realistic setting.
3. Realistic measurement resolution: The minimum time and space error considered are relative to the actor’s movement. The space resolution for a person is defined by the diameter of the body projection on the ground, which is set to 50 cm. The time resolution is defined by the time a person takes to walk a distance equal to the space resolution. In an indoor environment, considering a maximum speed of 1  m/s, the time resolution is 0.5  s.
4. Third quartile of point Euclidean error: The accuracy score is based on the third quartile of the error, which is defined as the 2D Euclidean distance between the measurement points and the estimated points for single-floor settings.

EvAAl Framework's extended criteria additionally adopted by the first EvAAL competitions are the following:

1. Secret path: The final path is disclosed immediately before the test starts, and only to the competitor whose system is under test. This prevents competitors from designing systems exploiting specific features of the path.
2. Independent actor: The actor is an agent not trained to use the localization system.
3. Independent logging system: The competitor system estimates the position twice per second, and sends the estimates to a logging application provided by the testing committee. This prevents any malicious actions from the competitors. The logging system is free software.
4. Identical path and timing: The actor walks along the same identical path with the same identical timing for all competitors, within time and space errors within the above-defined resolutions.

References

EvAAL competitions were organized yearly during the lifespan of the universAAL FP7 project, from 2011 to 2013. In 2014, the International Conference on Indoor Positioning and Indoor Navigation (IPIN) organised the first IPIN competition, building on EvAAL's experience. Since then, the IPIN conference has hosted a competition based on the EvAAL framework, with both on-site and off-site Tracks.

A concise definition of the EvAAL Framework and an introduction to how a competition is organised is provided in Section II "The EvAAL Framework" of Francesco Potortì, Antonino Crivello, Soyeon Lee and others, Offsite evaluation of localization systems: criteria, systems and results from IPIN 2021–22 competitions, IEEE Journal of Indoor and Seamless Positioning and Navigation, Vol. 2, p. 92–129, January 2024.

The EvAAL competitions were held from 2011 to 2013. They were based on the concept of competitive banchmarking, which is introduced and discussed in Paolo Barsocchi, Stefano Chessa, Francesco Furfari, and Francesco Potortì. Evaluating ambient assisted living solutions: the localization competition. IEEE Pervasive Computing, Vol. 12, No. 4, p. 72–79, October–December 2013. (preprint)

The EvAAL Framework was formally defined in 2014, based on the criteria of the EvAAL competition, as described in Francesco Potortì, Sangjoon Park, Antonio Ramòn Jiménez Ruiz and others, Comparing the performance of indoor localization systems through the EvAAL Framework. Sensors, Vol. 17, No. 10, October 2017.

The criteria are analyzed comprehensively in Antonino Crivello Francesco Potortì and Filippo Palumbo. Geographical and Fingerprinting Data for Positioning and Navigation Systems, Chap. 11 – The EvAAL Evaluation Framework and the IPIN Competitions, p. 209–224. Intelligent Data-Centric Systems, Elsevier, September 2018. (abstract)