Home Topics Schedule Submissions IEEE/RAS HRI Metrics Study Group Organizers
HRI 2020

HRI 2021 Workshop on Novel and Emerging Test Methods and Metrics for Effective HRI

HRI 2021 and all workshops will be held virtually this year. This full-day workshop was held virtually on 12 March, 2021. Below is the schedule for the workshop.

Please click here to view the workshop recording!

Tentative schedule (all times EST, GMT-5)
12:00 - Opening statements
12:15 - Invited talk, Andra Keay
13:00 - Short break
13:15 - Breakout rooms/Sessions
14:00 - Contributing Author - Conor McGinn (abstract)
14:10 - Contributing Author - Camille Peres (abstract)
14:20 - Contributing Author - Junxian Wang (abstract)
14:30 - Long break
15:00 - Keynote, Christoph Bartneck (recording)
16:00 - Short break
16:15 - Open discussion & IEEE SG presentation
17:00 - Adjourn

Despite large advances in robot interfaces and user-centric robot designs, the need for effective HRI continues to present challenges for the field of robotics. As new technologies are integrated into human-robot teams in a myriad of application domains ranging from smart manufacturing to home automation to space exploration, exposures to, and expectations from, robots are growing rapidly. A key barrier to achieving effective human-robot teaming in a multitude of domains is that there are few consistent test methods and metrics for assessing HRI effectiveness. The necessity for validated metrology is driven by the desire for repeatable and consistent evaluations of HRI methodologies. Such evaluations are critical for advancing the underlying theory of HRI, as well as establishing traceable mechanisms for vendors and consumers of HRI technologies to assess and assure functionality.

This workshop will address the issues surrounding the development of novel and innovative test methods and metrics for evaluating HRI performance across the multitude of application domains, including medical, field, manufacturing, and social robotics. This workshop is driven by the need for pushing the boundaries of testing and evaluating HRI, and to establish benchmarks and standards for advancing the state of the art of HRI performance. A key focus is on inter-disciplinary collaboration and multi-domain applicability of test methods and metrics. Specific goals include the following:

  • to develop and encourage the use of consistent test methods and metrics in evaluating HRI technologies, producing quality data sets of pragmatic applications, and validating human subject studies for HRI;
  • to explore novel and emerging metrology tools that have broad applicability across HRI domains;
  • to support a discussion about best practices in metrology and what features should be measured as the underlying theory of HRI advances;
  • to encourage the creation and sharing of high-quality, consistently-formatted datasets for HRI research; and
  • to promote the development of reproducible, metrics-oriented studies that seek to understand and model the human element of HRI teams.

Important Dates

  • 5 March 2021: Second submission deadline for extended abstracts
  • 8 March 2021: Notification of acceptance for workshop presentations
  • 12 March 2021: Full-day Workshop to take place virtually

Discussion Topics

Presentations by contributing authors will focus on the documentation of the test methods, metrics, and data sets used in their respective studies. Keynote and invited speakers will be selected from a targeted list of HRI researchers across a broad spectrum of application domains. Lightning talk participants will be selected from contributors reporting late-breaking evaluations and their preliminary results based on extended abstract submissions.

Discussions are intended to highlight the various approaches, requirements, and opportunities of the research community toward assessing HRI performance, enabling advances in HRI research, and establishing trust in HRI technologies. Specific topics of discussion will include:

  • reproducible and repeatable studies with quantifiable test methods and metrics;
  • human-robot collaboration and teaming test methods;
  • human data set content transferability, and traceability;
  • HRI metrics (e.g., situation and cultural awareness);
  • industry-specific metrology requirements.

Finally, this workshop is the third in a series of workshops leading toward formalized HRI performance standards. The IEEE Robotics and Automation Society (RAS) is hosting and supporting this standardization effort. Early workshops are intended to target community and consensus building, and on the establishment of a culture of repeatable and reproducible, metrology-based research in HRI. A fourth workshop is planned for the 2021 ACM/IEEE International Conference on Human Robot Interaction, and will specifically address the action items identified in this year's workshop. A workshop report documenting the presentations, discussions, and ensuing take-away and action items will be produced, and made publicly available.

Schedule

Confirmed Speakers (Schedule TBD):
Dr. Christoph Bartneck (University of Canterbury, HIT Lab NZ) - Keynote
Andra Keay (Managing Director and Founder of Silicon Valley Robotics, Visiting Scholar with CITRIS People and Robots) - Keynote

Accepted Submissions

Extended abstracts (up to 1p), detailing in-progress work outlining details of novel metrics and test methods for real-world HRI will be accepted for the workshop

Click here or visit https://forms.gle/vTS6WMZoBF3HPUR69 to submit to the workshop.

Organizers