Show simple item record

dc.contributor.authorUrkullu A.en_US
dc.contributor.authorPerez A.en_US
dc.contributor.authorCalvo B.en_US
dc.date.accessioned2019-05-02T17:19:49Z
dc.date.available2019-05-02T17:19:49Z
dc.date.issued2019-02-16
dc.identifier.issn1568-4946
dc.identifier.urihttp://hdl.handle.net/20.500.11824/975
dc.description.abstractIn many current problems, the actual class of the instances, the ground truth, is unavail- able. Instead, with the intention of learning a model, the labels can be crowdsourced by harvesting them from different annotators. In this work, among those problems we fo- cus on those that are binary classification problems. Specifically, our main objective is to explore the evaluation and selection of models through the quantitative assessment of the goodness of evaluation methods capable of dealing with this kind of context. That is a key task for the selection of evaluation methods capable of performing a sensible model selection. Regarding the evaluation and selection of models in such contexts, we identify three general approaches, each one based on a different interpretation of the nature of the underlying ground truth: deterministic, subjectivist or probabilistic. For the analysis of these three approaches, we propose how to estimate the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve within each interpretation, thus deriving three evaluation methods. These methods are compared in extensive experimentation whose empirical results show that the probabilistic method generally overcomes the other two, as a result of which we conclude that it is advisable to use that method when performing the evaluation in such contexts. In further studies, it would be interesting to extend our research to multiclass classification problems.en_US
dc.formatapplication/pdfen_US
dc.language.isoengen_US
dc.publisherApplied Soft Computingen_US
dc.relationES/1PE/SEV-2017-0718en_US
dc.relationES/1PE/TIN2017-82626-Ren_US
dc.relationEUS/BERC/BERC.2018-2021en_US
dc.relationEUS/ELKARTEKen_US
dc.rightsinfo:eu-repo/semantics/openAccessen_US
dc.rights.urihttp://creativecommons.org/licenses/by-nc-sa/3.0/es/en_US
dc.subjectModel selection, evaluation, crowdsourced data, AUC, Kendall-tauen_US
dc.titleOn the evaluation and selection of classifier learning algorithms with crowdsourced dataen_US
dc.typeinfo:eu-repo/semantics/articleen_US
dc.typeinfo:eu-repo/semantics/publishedVersionen_US
dc.identifier.doi10.1016/j.asoc.2019.02.019
dc.relation.publisherversionhttps://www.sciencedirect.com/science/article/pii/S1568494619300791en_US


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record

info:eu-repo/semantics/openAccess
Except where otherwise noted, this item's license is described as info:eu-repo/semantics/openAccess