Title: Bias in Human-in-the-loop Artificial Intelligence

Abstract: Paid micro-task crowdsourcing has gained popularity also thanks to the rise of AI because of the convenient way to generate large-scale manually annotated corpora and because of the possibility to create human-in-the-loop systems. However, when using crowdsourcing platforms for data gathering purposes, human factors need to be taken into account as humans now become part of (and are not just users of) the system.

In this talk I will discuss our recent research in the area of micro-task crowdsourcing with a focus on understanding crowd worker behaviors and their implications on the quality of the collected data and the bias in it. I will first discuss open challenges in the crowdsourcing ecosystem including issues caused by adversarial approaches that may disrupt the crowdsourcing model as we know it. I will then discuss how human bias is reflected in the data which is being collected by means of crowdsourcing. Finally, I will present our work making use of fine-grained behavioral logs.

Speaker Biography: Dr. Gianluca Demartini is an Associate Professor in Data Science at the University of Queensland, School of Information Technology and Electrical Engineering. His main research interests are Information Retrieval, Semantic Web, and Human Computation. His research has been supported by the Australian Research Council (ARC), the UK Engineering and Physical Sciences Research Council (EPSRC), and by the EU H2020 framework program. He received Best Paper Awards at the AAAI Conference on Human Computation and Crowdsourcing (HCOMP) in 2018 and at the European Conference on Information Retrieval (ECIR) in 2016, the Best Short Paper award at ECIR in 2020 and the Best Demo Award at the International Semantic Web Conference (ISWC) in 2011. He has published more than 100 peer-reviewed scientific publications including papers at major venues such as WWW, ACM SIGIR, VLDBJ, ISWC, and ACM CHI.

He has given several invited talks, tutorials, and keynotes at a number of academic conferences (e.g., ISWC, ICWSM, WebScience, and the RuSSIR Summer School), companies (e.g., Facebook), and Dagstuhl seminars. He is a senior member of the ACM since 2020, an ACM Distinguished Speaker since 2015, and has been a TEDx speaker in 2019.

He serves as co-editor in chief for the Human Computation journal, area editor for the Journal of Web Semantics, and editorial board member for the Information Retrieval journal. He is General co-Chair for the ACM International Conference on Information and Knowledge Management (CIKM) 2021. He has been Senior Program Committee member for, among others, the ACM Conference on Research and Development in Information Retrieval (SIGIR), the ACM Web Search and Data Mining (WSDM) Conference, the AAAI Conference on Human Computation and Crowdsourcing (HCOMP), and the International Conference on Web Engineering (ICWE). He is Program Committee member for several conferences including WWW, SIGIR, KDD, IJCAI, AAAI, ISWC, and ICWSM. He was Crowdsourcing and Human Computation Track co-Chair at WWW 2018 and co-chair for the Human Computation and Crowdsourcing Track at ESWC 2015. He co-organized several workshops and tutorials at international conferences as well as the Entity Ranking Track at the Initiative for the Evaluation of XML Retrieval in 2008 and 2009.

Before joining the University of Queensland, he was Lecturer at the University of Sheffield in UK, post-doctoral researcher at the eXascale Infolab at the University of Fribourg in Switzerland, visiting researcher at UC Berkeley, junior researcher at the L3S Research Center in Germany, and intern at Yahoo! Research in Spain. In 2011, he obtained a Ph.D. in Computer Science at the Leibniz University of Hanover focusing on Semantic Search.

Homepage: http://www.gianlucademartini.net/

RSVP: https://www.academicfringe.org/registration