Technical Annex on-site

Status of this document

This version is intended to give a complete overview on the criteria used to organise the track 1 (Smartphone based positioning) and track 2 (Foot-mounted pedestrian dead reckoning positioning) competitions and evaluate the competing systems. Some detailed information is still missing, as indicated in specific points of this document. This version of the technical annex will be augmented with the missing details and with additional information provided by the organisers, possibly with the help of feedback from the competitors.

Addenda to this document are communicated via the contest(at)evaal.aaloa.org mailing list and subsequently added to the last section Clarifications and Additions. Please use the list in order to ask for more information.

Overview of the competition

The purpose of the on-site competition is to assess and measure the ability of competing systems to accurately identify their position inside a large, public indoor area. Competing system should be engineered or prototypal systems intended to be carried by an actor without imparing her movements. Competitors will be provided a detailed map of the area, while the predifined path followed by the actor will not be disclosed to competitors before the application of the benchmarks and my span multiple buildings. A competing system, while carried by the actor, will continuously communicate real-time estimates of its position to a measurement app provided by IPIN. Final scores will be based on the accuracy of the estimates as measured by the dedicated app. For track 1 (Smartphone based), competitors can use any sensor available on the smartphones used. Only one commercially available smartphone per competitor can be used in order to localise the actor. For track 2 (Foot-mounted pedestrian dead reckoning), each team should implement a localisation system that exploits MEMS sensors (such as inertial, compass and pressure sensors), without limits on the number of devices, which must be mounted on the feet, no higher than the ankle articulations. No instrumentation of the area by competitors is allowed. The only information the competing systems can gather on the environment is the presence of Wi-Fi spots installed in the area. Competitors are requested to integrate their solution with the measurement app provided by the EvAAL organizer. The EvAAL team will assist the competitors in this phase and a competitor's integration package will be delivered for this purpose. Details about integration will be provided as part of the competitor's integration package. Straps or any other devices necessary for the actor to carry the smartphone (for track 1) and the sensors (for track 2) should be provided by the competitors. Possible specific requirements of the proposed localisation system should be communicated at an early stage in order be approved and to make the necessary arrangements. For any technical inquiries please e-mail the competition chairs and the TPC chair.

Measurement procedure

The score for each competing artefact will be evaluated in the course of the time slot assigned to each competitor. At the beginning of the time slot, the competing team will configure their artefact and deploy it on the actor's body. Subsequently, the actor will start moving and the measurement will take place; during this phase the competitors will have the opportunity to perform only short reconfigurations of their systems, in the order of few seconds. The actor walks at a natural pace along a loosely-defined reference path, equal for all competitors. The path connects some tens of keypoints . When the actor passes through these keypoints, the actor will set a time mark using the measurement app. The list of time marks together with the ID and positions of the keypoints will be the ground truth used to compute the localisation errors. The competing localisation systems should provide coordinates with a suggested frequency of 2 Hz to the measurement app, but only the last estimate prior to each time mark will be taken into account to evaluate the competing system accuracy. The path followed by the person will be approximately the same for each test, will take approximately the same time and will pass through all the keypoints in the same order. It may include pauses, loops and any kind of natural movement. It will be disclosed to competitors few days before the competition.

Evaluation criterion

The accuracy score will be computed as a monotone increasing function of the third quartile of the localisation errors at the keypoints. The localisation error is the Euclidean distance between the competitor's estimate and the real position of a keypoint. As far track 2 is concerned, estimation errors will receive a penalty depending on the number of devices mounted on the feet. Final scores will be disclosed at the end of the competition, and the competing systems ranked according to this final score. Additional details on the evaluation criteria will be provided as part of this document.

Clarifications and additions

This section summarises communications made on the contest mailing list from May to September 2015.

We suggest a frequency of 2 Hz for sending data to the measurement app, but there is no hard requirement on the updating rate.  A frequency that is too high may in principle slow down the measurement application, and indirectly harm the precision of your app.  A frequency that is too low may be the cause of a bigger estimation error, as estimates are done when the actor presses the button, and they will refer to the last position received by the app before the button is pressed.

Coordinates will need to be in the WGS84 coordinate system (longitude and latitude) for x, y, and the number of floor (an integer starting from 0) for z.  The timestamp should be in milliseconds from the epoch, retrieved from currentTimeMillis().

The coordinates of the starting point for the path walked by the actor will be provided the day of the competition, at least half an hour before the competition starts.  We will install about 20 BLE beacons in the area (see estimote.com/). You will have at least a full day before the competition to survey the area yourself, take measurements where needed and make measurements of the WiFi and BLE signals.   The area will span two or three buildings, very close to each other, so a small part of the path will be outdoor.

A number of markers will be put on the floor.  The actor will walk a path passing by all markers keeping in hand the phone on which the measurement app runs, usually looking at the screen, because a button must be pressed when passing over the markers on the floor.

The actor will press a button on the measurement app each time he passes over on marker.  We estimate that a trained actor will provide marker timestamps with an error less than 250 ms in time and less than 0.5 m in space.  While walking the path, actors will sometimes make an error, like forgetting to press a button: in this case the test will be stopped and repeated from the start.  You are advised to follow the actor during the path, so you can check that everything is well. You cannot tune your application after the official start of the competition.  However, if you notice that things are clearly going wrong in your app (crashing, for example), you may ask for a second chance, which will be normally given if time permits.  In any case, the path will be run twice for all competitors, and the best result will be retained.

Competitors in track 1 can use any device available on their commercial smartphone (not tablet), and nothing more. The actor will keep the phone in hand all the time,  generally with the screen looking upwards, but without any guarantee of a precise position.

Competitors in track 2 can use MEMS sensors mounted on the feet, and nothing more.  They can use a smartphone or tablet to control, monitor and process the data coming from the foot-mounted devices.

All competitors in tracks 1 & 2 are required to integrate their app with the dedicated app that we provide. Please do so as soon as possible so that we can solve possible problems well in advance of the competition. Your phone will be running your application as a service, while the measurement app will run in the foreground.

The error will be measured based on xy coordinates (longitude and latitude).  To this, a penalty P=15m will be added for each floor error. For example, if the xy error is 4m and the estimated floor is 2 while it should be 0, the computed error for that estimate will be 4+2*P=34m.

Competitors for which the third quartile of error is greater than 25m are not eligible for the winner prize.

Have a look at the maps and the path simulation.