Scoring and Standings 2017/2018

The ERL Emergency Service Robots is composed of one major tournament, held in Piombino 15-23 September 2017. Teams participate and get scores based on their performances. A final score is computed for each team, per Task Benchmark (TBM) and Functionality Benchmark (FBM), using the best run of the trial. Teams are ranked based on their final score.

More information can be found on ERL Emergency Rulebook.  

Prizes for the top teams per TBM were awarded during the Awards Ceremony held at Giardini Pro Patria, in Piombino, Italy; and prizes for the best teams per FBM were awarded during the European Robotics Forum (ERF) in Tampere, 13-15 March 2018.

The final scoring and standings for the ERL Emergency Service Robots Challenge  are the ones below:

 

Task Benchmarks

TBM1: The Grand Challenge (Land +Sea + Air)

Sorted Based on best run

Team

A

P

Telerob + Universitat de Girona + ISEP/INSESC TEC

67

0

Robdos-IMM + IIS Piombino CVP

46

0

Raptors + Oubot

40

0

Tuscany Robotics

38

3

Bebot + AUV TomKyle + HSR Search and Rescue

33

2

A - Achievements

P – Penalties

Maximum achievements: 98

ERL Emergency 2017 - TBM1-results

 

TBM2: Survey the building and search for missing workers (Land +Air)

Sorted Based on best run

Team

A

P

IMM + IIS Piombino CVP

23

0

Raptors

21

0

Telerob + ISEP/INSESC TEC

19

0

ENSTA Team

16

1

ENSTA Bretagne

13

1

Bebot + HSR Search and Rescue

8

2

ETH Zürich ASL-RSL

4

2

Maximum achievements: 40

ERL Emergency 2017 - TBM2 -results

 

TBM3: Pipe inspection and search for missing workers (Sea +Air)

Sorted Based on best run

Team

A

P

Universitat de Girona + ISEP/INSESC TEC

24

0

Tuscany Robotics

16

0

AUV TomKyle + HSR Search and Rescue

16

1

Robdos + IIS Piombino CVP

14

0

Raptors + Oubot

9

0

ENSTA Bretagne

6

0

ENSTA Team

0

0

 Maximum achievements: 64

ERL Emergency 2017 - TBM3- results

 

TBM4: Stem the leak (Land +Sea)

Sorted Based on best run

Team

A

P

Telerob + Universitat de Girona

24

0

Raptors + Oubot

17

0

Bebot + AUV TomKyle

11

3

Robdos-IMM

10

0

ENSTA Team

7

2

ENSTA Bretagne

6

1

Maximum achievements: 58

ERL Emergency 2017 - TBM4 - results

  

Functionality Benchmarks

FBM1: Mapping Functionality (Land + Air)

Team RMSE Ground coverage   Score
IMM (Land)+ IIS Piombino CVP (Air) 1 0.50 0.75
ENSTA Bretagne (Land+Air) 1 0 0.50
Telerob (Land)+ISEP/INSESC TEC (Air) 0.91333 0 0.46
ETH (Land+Air) 0.82667 0 0.42
Bebot (Land) + HSR Search and Rescue (Air) 0 0 0
ENSTA Team(Land+Air) 0 0 0
Raptors (Land+Air) 0 0 0

Maximum Score: 1

SCORE=(RMSE+ Ground coverage)/2

ERL Emergency 2017 - FBM1 -results

 

FBM2: Object Recognition (Land + Air)

Sorted based on best run
Team OPIs found  Precision Recall Index
Telerob (Land)+ISEP/INSESC TEC (Air) 9 1 0.81818 0.90000
Raptors (Land+Air) 8 1 0.72727 0.84211
IMM (Land)+ IIS Piombino CVP (Air) 7 1 0.63636 0.77778
ENSTA Team(Land+Air) 5 1 0.45455 0.62500
ENSTA Bretagne (Land+Air) 4 1 0.36364 0.53333
Bebot (Land) + HSR Search and Rescue (Air) 1 1 0.09091 0.16667
ETH (Land+Air) 0 0 0 0

Maximum Index: 1

Index= F-mesure = 2*(Precission/Recall)/(Precision+Recall)

for Beta=1

(See Rulebook for equations).

ERL Emergency 2017 - FBM2 - results

 

FBM3: Object Detection (Sea)

Sorted based on best run
Team  Buoy 1 
 Buoy 2 
 Buoy 3  Buoy 4
 Buoy 5
 CI   CC 
 Score
Universitat de Girona Yes Yes No Yes Yes 0 4 4
AUV Team Tomkyle Yes Yes No Yes No 0 3 3
Tuscany Robotics Team No No Yes No Yes 0 2 2
ENSTA Bretagne Yes No No No No 0 1 1
ENSTA Team No No No No No 0 0 0
OUBOT No No No No No 0 0 0
Robdos No No No No No 0 0 0

CI - Correctly Identified buoys (via the black number)

CC – Correctly Classified buoys (without the detection of the black number)

SCORE=2.5*CI+CC

ERL Emergency 2017 - FBM3 - results

  

FBM4: Vertical Wall Mapping Functionality (Air)

Team RMSE MaxErr Index
IIS Piombino CVP 0.31 18 0.98
ISEP/INESC TEC 0.45 18 0.97
HSR Search and Rescue 13.41 18 0.25
Tuscany Robotics Team - - 0
RAPTORS - - 0

Maximum Index: 1

ERL Emergency 2017 - FBM4 -results