Detailed Evaluation Criteria

The score is given by:

Accuracy score = 10*T+5*U

EC1.2:For second phase, the stream produced by competing systems will be compared against a log file of the expected position of the user. Specifically, we will evaluate the individual error of each measure (the Euclidian distance between the measured and the expected points), and we will determine the 75^th percentile P of the errors. P will be scaled according to the following formula:

Accuracy score =

10                              if          P<=0,5m
4*(0.5-P)+10               if          0,5m < P <= 2 m
2*(4-P)                       if          2 m < P <= 4 m
0                               if          P > 4 m

The final score on accuracy will be the average between the scores obtained in all the phases.

EC1.3:The score for the third phase will be calculated according to EC1.2

EC2 Installation complexity [weight 0.10]

– effort required to install the AAL localization system in a flat, measured by the evaluation committee as a function of the person-minutes of work needed to complete the installation (The person-minutes of the first installer will be fully included; the person-minutes of any other installer will be divided by 2)

The time T is measured in minutes from the time when the competitors enter the living lab to the time when they declare the installation complete (no further operations/configurations of the system will be admitted after that time), and it will be multiplied by the number of people N working on the installation. The parameters will be translated in a score (ranging from 0 to 10) according with the following formula:

Installation Complexity 

10                                    if               T*N <=10

10 * (60-T*N) / 50              if               10

EC3. User acceptance [weight 0.2]

– expresses how much the localization system is invasive in the user’s daily life and thereby the impact perceived by the user; this parameter will be computed by evalauting a set of statements with the following possible answers, associated to a value:

• 4: "true already now"
• 3: "not true, but we are already working on it"
• 2: "no, but it can be done"
• 1: "no it can't"

If the  statement cannot be evaluated the maximum scoring will be given (i.e. 4).

For each statement the values related to the answer are averaged with a weight. The statements are grouped into the following categories: 

• Tags: does the system require wearing tags?
• Environment: does the system affect home environment?
• Presence: is the system felt as "always there"?
• Maintenance: is the system easy to be maintained?

The following table shows the statements and the related weight.

For each category the weighted answers are averaged and normalized in a 0 to 10 scale, then the average of the 4 categories is given as final score.

EC4 Availability [weight 0.2]

 – fraction of time the localization system was active and responsive. It is measured as the ratio between the number P of produced localization samples and the number of expected samples E: competing systems are expected to provide one sample every half a second. Excess samples are discarded.

The values of availability A will be translated into a score (ranging from 0 to 10) according to the following formula:

Availability score = 10 *P/E

EC5 Interoperability with AAL systems [weight 0.15]

 –evaluates the degree of interoperability of the solution in terms of openness of the SW, adoption of standards for both SW and HW, replaceability of parts of the solution with other ones. As with EC3, this parameter will be computed by evalauting a set of statements with the following possible answers, associated to a value:

• 4: "true already now"
• 3: "not true, but we are already working on it"
• 2: "no, but it can be done"
• 1: "no it can't"

If the  statement cannot be evaluated the maximum scoring will be given (i.e. 4).

For each statement the values related to the answer are averaged with a weight. The statements are grouped into the following categories:

• Code integration: is the system easy to be integrated in other SW?
• Testing and configuration: is the system easy to be configured and tested?
• Portability: can the system be ported to different platforms OSes?
• HW: are the specifications of HW available, and the firmware?

The following table shows the statements and the related weight.

For each category the weighted answers are averaged and normalized in a 0 to 10 scale, then the average of the 4 categories is given as final score.