Detailed Evaluation Criteria

EC1 Accuracy [weight 0.35]

– each produced localization sample is compared with the reference position and the error distance is computed. EC1 will be calculated by the arithmetic mean of EC1.1, EC1.2 and EC1.3.

EC1.1:During the first phase of the competition the user will stop (after a predefined walk equal for all competitors) 30 seconds in each Area of Interest (AoI). Accuracy in this case will be measured as the fraction T of time in which the localization system provides the correct information about:

  • Presence of the user in a given AoI
  • Absence of the user from any AoI

For what concerns the accuracy, the score evaluated during the phase 1 (AoI benchmark), we give half a point for the fraction U of time the system confuses between a big and a small areas. For instance, in the cases (illustrated in the figure):

  • system indicates AOI4, but really is AOI41
  • system indicates AOI41, but really is AOI4

The score is given by:

Accuracy score = 10*T+5*U

EC1.2:For second phase, the stream produced by competing systems will be compared against a log file of the expected position of the user. Specifically, we will evaluate the individual error of each measure (the Euclidian distance between the measured and the expected points), and we will determine the 75th percentile P of the errors. P will be scaled according to the following formula:

Accuracy score =

10                              if          P<=0,5m
4*(0.5-P)+10               if          0,5m < P <= 2 m
2*(4-P)                       if          2 m < P <= 4 m
0                               if          P > 4 m

The final score on accuracy will be the average between the scores obtained in all the phases.

EC1.3:The score for the third phase will be calculated according to EC1.2

 

EC2 Installation complexity [weight 0.10]

– effort required to install the AAL localization system in a flat, measured by the evaluation committee as a function of the person-minutes of work needed to complete the installation (The person-minutes of the first installer will be fully included; the person-minutes of any other installer will be divided by 2)

The time T is measured in minutes from the time when the competitors enter the living lab to the time when they declare the installation complete (no further operations/configurations of the system will be admitted after that time), and it will be multiplied by the number of people N working on the installation. The parameters will be translated in a score (ranging from 0 to 10) according with the following formula:

Installation Complexity 

10                                    if               T*N <=10

10 * (60-T*N) / 50              if               10

 

EC3. User acceptance [weight 0.2]

– expresses how much the localization system is invasive in the user’s daily life and thereby the impact perceived by the user; this parameter will be computed by evalauting a set of statements with the following possible answers, associated to a value:

  • 4: "true already now"
  • 3: "not true, but we are already working on it"
  • 2: "no, but it can be done"
  • 1: "no it can't"

If the  statement cannot be evaluated the maximum scoring will be given (i.e. 4).

For each statement the values related to the answer are averaged with a weight. The statements are grouped into the following categories: 

  • Tags: does the system require wearing tags?
  • Environment: does the system affect home environment?
  • Presence: is the system felt as "always there"?
  • Maintenance: is the system easy to be maintained?

The following table shows the statements and the related weight.

 

SentenceWeight
Tags 
The system does NOT need the user to bring something with him all the time to be localized.5
This "thing" is easily wearable.2
Environment 
The installation is well hidden in the house (e.g. nodes are <5 and not bigger than a hand plam).5
The installation does NOT need considerably more cabling than a typical Home PC installation (including for example printer, scanner, phone connector etc.).4
The installation can be provided as a piece of forniture.2
Presence 
The system does NOT use video cameras.3
The system does NOT require line of sight (no obstacles in the middle).2
Maintenance 
The user does NOT need to replace/charge batteries in some device.4
The user needs to replace/charge batteries in more than three months.5
The user does NOT need to re-calibrate the system (e.g. when the furniture changes).4
The user needs to re-calibrate the system in more than three months.5

For each category the weighted answers are averaged and normalized in a 0 to 10 scale, then the average of the 4 categories is given as final score.

EC4 Availability [weight 0.2]

 – fraction of time the localization system was active and responsive. It is measured as the ratio between the number P of produced localization samples and the number of expected samples E: competing systems are expected to provide one sample every half a second. Excess samples are discarded.

The values of availability A will be translated into a score (ranging from 0 to 10) according to the following formula:

Availability score = 10 *P/E

 

EC5 Interoperability with AAL systems [weight 0.15]

 –evaluates the degree of interoperability of the solution in terms of openness of the SW, adoption of standards for both SW and HW, replaceability of parts of the solution with other ones. As with EC3, this parameter will be computed by evalauting a set of statements with the following possible answers, associated to a value:

  • 4: "true already now"
  • 3: "not true, but we are already working on it"
  • 2: "no, but it can be done"
  • 1: "no it can't"

If the  statement cannot be evaluated the maximum scoring will be given (i.e. 4).

For each statement the values related to the answer are averaged with a weight. The statements are grouped into the following categories:

  • Code integration: is the system easy to be integrated in other SW?
  • Testing and configuration: is the system easy to be configured and tested?
  • Portability: can the system be ported to different platforms OSes?
  • HW: are the specifications of HW available, and the firmware?

The following table shows the statements and the related weight.

 

SentenceWeight
Code integration 
An API is provided for integrating the system into others.5
Written documentation is provided for this API (e.g. documents, wiki, tutorials, samples).4
In code documentation is provided (e.g. javadoc).3
The SW is publicly available for free.4
The SW is licensed as open source.5
A well knwon protocol for remote operation is given that does not require further specification at app level.5
Testing and configuration 
Tools for testing/monitoring the system are provided.4
The tool is graphical an simple to use.2
A tool for configuring/calibrating the system is given.>4
The tool is graphical an simple to use.2
Portability 
The system can run on the following OSes: 
Windows3
Unix/linux3
MacOS3
Android3
Others3
Part of the SW can be substituted by another existing one (e.g. other existing libraries).3
HW 
Parts of the system can be substituted by another commercially available one WITHOUT ANY modification on the HW and/or the firmware.5
The used firmware is available for free.3
The firmware is opensource.2
The HW specifications are publicly available.3
The HW is Open Source HW.2

 

For each category the weighted answers are averaged and normalized in a 0 to 10 scale, then the average of the 4 categories is given as final score.