Automation of Scientific Reviewer Assignment: a Survey of Problems and Methods
DOI:
https://doi.org/10.31649/1997-9266-2024-172-1-56-64Keywords:
peer review, reviewing, eviewer assignment problem, natural language processing, general assignment problem, discrete optimization, similarity metric, topic modeling, language modelsAbstract
High-quality and timely peer review for scientific works has become increasingly acute recently. Scientific reviewers peer review journal articles, conference papers, monographs, PhD-thesis, grants etc. Scientific reviewers are assigned mainly manually due to the lack of time. Having large volumes of content to review, a high-quality peer review is hard to get. In recent years there has been an increase in research on reviewer assignment automation. Also, the formal comparison of a reviewer’s research domains and a proposal’s research domains, or character-level comparison of keywords do not always provide high-quality assignments. In this paper, we perform a review on current methods of automated scientific reviewer assignment. There are 3 stages of reviewer assignment process: 1) creating reviewers’ database and structuring the information about reviewers and proposals; 2) calculating the similarity score between a proposal and a reviewer; 3) finding the appropriate assignment of proposals to reviewers, and maximizing the aggregated similarity and overall topic coverage over all assignments. Two main variations of reviewer assignment problem are considered: single and multiple reviewer assignment problem. Methods for structuring the information about proposals and reviewers based on statistical analysis of text, topic modeling and deep learning are analyzed. In the third stage, we considered possible optimal criteria for optimizing reviewers’ assignments to proposals, and also constraints, that ensure a certain level of reviewers-proposal concordance such as overall topic coverage, fair peer review, reviewers workload etc. The problem of optimal reviewer assignment is NP-complete, therefore for solving different heuristics and meta-heuristics algorithms are used. We also present perspectives of future research on the automated reviewer assignment.
References
N. B. Shah, “An Overview of Challenges, Experiments, and Computational Solutions in Peer Review,” Communications of the ACM, vol. 65, no. 6, pp. 76-87, June 2022 https://doi.org/10.1145/3528086 .
S. Price, and P. A. Flach, “Computational Support for Academic Peer Review: A Perspective from Artificial Intelligence,” Communications of the ACM, 2017.
T. Brown, “Peer Review and the Acceptance of New Scientific Ideas: Discussion Paper from a Working Party on Equipping the Public with an Understanding of Peer Review,” Sense About Science, November 2002-May 2004.
А. Т. Ашеров, Диссертации. Экспресс-анализ качества: руководство для экспертов, оппонентов и науч. руководителей, моногр. Украинская инженерно-педагогическая академия. Харків: Кортес-2001, 2008, 53 с.
D. J. Benos, E. Bashari, et al. “The ups and downs of peer review,” Adv Physiol Educ., no. 31(2), pp. 145-152, 2007. https://doi.org/10.1152/advan.00104.2006 .
T. Jefferson, P. Alderson, E. Wager, and F. Davidoff, “ Effects of editorial peer review: a systematic review,” JAMA, no. 287(21), pp. 2784-2786, 2002. https://doi.org/10.1001/jama.287.21.2784 .
L. Bornmann, R. Haunschild, and R. Mutz, “Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases,” Humanit Soc Sci Commun, no. 8, pp. 224, 2021. https://doi.org/10.1057/s41599-021-00903-w .
S. T. Dumais, and J. Nielsen, “Automating the assignment of submitted manuscripts to reviewers,” in Proceedings of the Fifteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 1992, pp. 233-244. Publ by ACM. https://doi.org/10.1145/133160.133205 .
A. C. Ribeiro, A. Sizo, and L. P. Reis, “Investigating the reviewer assignment problem: A systematic literature review,” Journal of Information Science, 2023. https://doi.org/10.1177/01655515231176668 .
Meltem Aksoy, Seda Yanik, and Mehmet Fatih, Amasyali. Reviewer Assignment Problem: A Systematic Review of the Literature. J. Artif. Int. Res. 76, May 2023. https://doi.org/10.1613/jair.1.14318 .
F. Wang, N. Shi, and B. Chen, “A comprehensive survey of the reviewer assignment problem,” International Journal of Information Technology and Decision Making, no. 9 (4), pp. 645-668, 2010. https://doi.org/10.1142/S0219622010003993 .
F. Wang, B. Chen, and Z. Miao, “A survey on reviewer assignment problem,” Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5027 LNAI(60704048), pp. 718-72, 2008. https://doi.org/10.1007/978-3-540-69052-8_75 .
S. Tan, Z. Duan, S. Zhao, J. Chen, and Y. Zhang, “Improved reviewer assignment based on both word and semantic features,” Information Retrieval Journal, no. 24 (3), pp. 175-204, 2021. https://doi.org/10.1007/s10791-021-09390-8 .
D. Yarowsky, and R. Florian, “Taking the load off the conference chairs: Towards a digital paper-routing assistant,” in Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, EMNLP 1999, pp. 220-230, 1999. Association for Computational Linguistics (ACL).
M. Karimzadehgan, C. X. Zhai, and G. Belford, “Multi-aspect expertise matching for review assignment,” in Proceedings of International Conference on Information and Knowledge Management, 2008, pp. 1113-1122, https://doi.org/10.1145/1458082.1458230 .
M. Mirzaei, J. Sander, and E. Stroulia, “Multi-Aspect Review-Team Assignment using Latent Research Areas,” Information Processing and Management, no. 56 (3), pp. 858-878, 2019. https://doi.org/10.1016/j.ipm.2019.01.007 .
D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,” Journal of Machine Learning Research, no. 3 (4-5), pp. 993-1022, 2003. https://doi.org/10.7551/mitpress/1120.003.0082 .
E. Ekinci, and S. I. Omurca, “NET-LDA: A novel topic modeling method based on semantic document similarity,” Turkish Journal of Electrical Engineering and Computer Sciences, no. 28 (4), pp. 2244-2260,2020. https://doi.org/10.3906/ELK-1912-62 .
O. Anjum, H. Gong, S. Bhat, J. Xiong, and W. M. Hwu, “Pare: A paper-reviewer matching approach using a common topic space,” in EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pp. 518-528, 2019. Association for Computational Linguistics. https://doi.org/10.18653/v1/d19-1049 .
С. Д. Штовба, М. В. Петричко, і М. Ю. Петранова, «Метрика схожості категоріальних розподілів, що враховує спорідненість різних категорій,» Вісник Вінницького політехнічного інституту, № 2 (167), с. 49-57, 2023. https://doi.org/10.31649/1997-9266-2023-167-2-49-57 .
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations ofwords and phrases and their compositionality,” in Advances in Neural Information Processing Systems. Neural information processing systems foundation, 2013.
J. Pennington, R.Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in EMNLP 2014–2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 2014, pp. 1532-1543. Association for Computational Linguistics (ACL). https://doi.org/10.3115/v1/d14-1162 .
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov,. “Enriching Word Vectors with Subword Information,” Transactions of the Association for Computational Linguistics, no. 5, pp. 135-146, 2017.
J. Howard, and S.Ruder, “Universal Language Model Fine-tuning for Text Classification.” ACL, Association for Computational Linguistics. 2018. http:// arxiv.org/abs/1801.06146 .
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv Preprint. arXiv:1810.04805v2. 1-16. 2018.
R. Alec, W. Jeffrey, C. Rewon, L. David, A. Dario, and S. Ilya, “Language Models are Unsupervised Multitask Learners | Enhanced Reader,” OpenAI Blog, no. 1(8), p. 9, 2019.
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108. 2019.
C. Sun,, K. Ng, T. J. Henville, P., and R. Marchant, “Hierarchical word mover distance for collaboration recommender system, in Communications in Computer and Information Science, vol. 996, pp. 289-302, 2019. Springer Verlag. https://doi.org/10.1007/978-981-13-6661-1_23 .
X. Kong, H. Jiang, Z. Yang, Z. Xu, F. Xia, and A. Tolba, “Exploiting Publication Contents and Collaboration Networks for Collaborator Recommendation,” PLoS ONE , no. 11(2): e0148492. 2016. https://doi.org/10.1371/journal.pone.0148492 .
Y. Zhao, J. Tang, and Z. Du, “EFCNN: A Restricted Convolutional Neural Network for Expert Finding,” in: Yang Q., Zhou Z. H., Gong Z., Zhang M. L., Huang S. J., Eds Advances in Knowledge Discovery and Data Mining. PAKDD 2019. Lecture Notes in Computer Science, vol. 11440, 2019. Springer, Cham. https://doi.org/10.1007/978-3-030-16145-3_8 .
D. G. Cattrysse, and L. N. Van Wassenhove, “A survey of algorithms for the generalized assignment problem,” European Journal of Operational Research,no. 60 (3), pp. 260-272, 1992. https://doi.org/10.1016/0377-2217(92)90077-M .
C. Long, R. C. W. Wong, Y. Peng, and L. Ye, “On good and fair paper-reviewer assignment,” in Proceedings IEEE International Conference On Data Mining, 2013, pp. 1145-1150. https://doi.org/10.1109/ICDM.2013.13 .
J. Goldsmith, and R. H. Sloan, “The AI conference paper assignment problem,” in AAAI Workshop – Technical Report, vol. WS-07-10, pp. 53-57, 2007.
W. D.Cook, B. Golany, M. Kress, M. Penn, and Raviv, “Optimal allocation of proposals to reviewers to facilitate effective ranking,” Management Science, no. 51 (4), pp. 655-661, 2005. https://doi.org/10.1287/mnsc.1040.0290 .
T. Kolasa, and D. Krol, “A survey of algorithms for paper-reviewer assignment problem,” IETE Technical Review (Institution of Electronics and Telecommunication Engineers, India, no. 28 (2), pp. 123-134, 2011. https://doi.org/10.4103/0256-4602.78092 .
L. Charlin, R. Zemel, and C. Boutilier, “A framework for optimizing paper matching,” in Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, pp. 86–95, 2012.
L. Charlin, and R. S. Zemel, “The Toronto Paper Matching System: An automated paper-reviewer assignment system,” in ICML Workshop on Peer Reviewing and Publishing Models (PEER), vol. 28, 2013.
D. K. Tayal, P. C. Saxena, A. Sharma, G. Khanna, and S. Gupta, “New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions,” Applied Intelligence, no. 40 (1), pp, 54-73, 2014. https://doi.org/10.1007/s10489-013-0445-5 .
M. Karimzadehgan, and C. X. Zhai, “Integer linear programming for Constrained Multi-Aspect Committee Review Assignment,” Information Processing and Management, no. 48 (4), pp. 725-740, 2012. https://doi.org/10.1016/j.ipm.2011.09.004 .
M. Karimzadehgan, and C. X. Zhai, “|Constrained multi-aspect expertise matching for committee review assignment,” in International Conference on Information and Knowledge Management, Proceedings, 2009, pp. 1697-1700. https://doi.org/10.1145/1645953.1646207 .
W. Tang, J. Tang, T. Lei, C. Tan, B. Gao, and T. Li, “On optimization of expertise matching with various constraints,” Neurocomputing, no. 76 (1), pp. 71-83, 2012. https://doi.org/10.1016/j.neucom.2011.04.039 .
W. Tang, J. Tang, and C. Tan, “Expertise matching via constraint-based optimization,” in Proceedings–2010 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2010, vol. 1, pp. 34-41, 2010. https://doi.org/10.1109/WI-IAT.2010.133 .
N. Xue, J. X. Hao, S. L. Jia, and Q. Wang, “An interval fuzzy ontology-based peer review assignment method,” in Proceedings of IEEE International Conference on E-Business Engineering, 2012, 55-60. https://doi.org/10.1109/ICEBE.2012.19 .
A. Kale, R. Kharat, S. Bodkhe, P. Apte, and H. Dhonde, “Automated fair paper reviewer assignment for conference management system,” in Proceedings of 1st International Conference on Computing, Communication, Control and Automation, ICCUBEA 2015, 2015, pp. 408-411. https://doi.org/10.1109/ICCUBEA.2015.85 .
D. Hartvigsen, J. C. Wei, and R. Czuchlewski, “The conference paper-reviewer assignment problem” Decision Sciences, vol. 30, pp. 865-876, 1999. Decision Sciences Institute. https://doi.org/10.1111/j.1540-5915.1999.tb00910.x .
N. Garg, T. Kavitha, A. Kumar, K. Mehlhorn, and J. Mestre, “Assigning papers to referees,” Algorithmica (New York), no. 58 (1), pp. 119-136, 2010. https://doi.org/10.1007/s00453-009-9386-0.
I. Stelmakh, N. B. Shah, and A. Singh, “PeerReview4All: Fair and Accurate Reviewer Assignment,” in Peer Review. arXiv preprint, arXiv:1806.06237, 2019. https://arxiv.org/abs/1806.06237 .
A. Kobren, B. Saha, and A. McCallum, “Paper matching with local fairness constraints,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, 2019, pp. 1247-1257, New York, USA. ACM.
J. Pay and Y. Zick, “I Will Have Order!” Optimizing Orders for Fair Reviewer Assignment. arXiv preprint, arXiv:2108.02126, 2021. https://doi.org/10.48550/arXiv.2108.02126.
K. Dhull, S.J ecmen, P. Kothari, and N. B. Shah, “Strategyproofing Peer Assessment via Partitioning,” The Price in Terms of Evaluators’ Expertise. arXiv preprint, arXiv:2201.10631, 2022. https://doi.org/10.48550/arXiv.2201.10631 .
N. Alon, F. Fischer, A. Procaccia, and M. Tennenholtz, “Sum of us: Strategyproof selection from the selectors,” in Proceedings of the 13th Conference on Theoretical Aspects of Rationality and Knowledge, 2011, pp. 101-110.
S. Jecmen, H. Zhang, R. Liu, N. B. Shah, V. Conitzer, and F. Fang, “Mitigating Manipulation,” in Peer Review via Randomized Reviewer Assignments. arXiv preprint arXiv:2006.16437, 2020. http://arxiv.org/abs/2006.16437 .
L. Guo, J. Wu, W. Chang, J. Wu, and J. Li, “K-Loop Free Assignment,” in Conference Review Systems. In 2018 International Conference on Computing, Networking and Communications, ICNC 2018, 2018, pp. 542-547. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ICCNC.2018.8390343 .
N. Boehmer, R. Bredereck, and A. Nichterlein, “Combating Collusion Rings is Hard but Possible,” arXiv preprint, arXiv:2112.08444, 2021. https://doi.org/10.48550/arXiv.2112.08444 .
K. Leyton-Brown, Mausam, Y. Nandwani, H. Zarkoob, C. Cameron, N. Newman, and D. Raghu, “Matching Papers and Reviewers at Large Conferences,” arXiv preprint, arXiv:2202.12273, 2022. https://arxiv.org/abs/2202.12273 .
T. Silva, M. Jian, and Y. Chen, “Process analytics approach for R&D project selection.” ACM Transactions on Management Information Systems, no. 5(4), 2014. https://doi.org/10.1145/2629436 .
C. J. Taylor, “On the optimal assignment of conference papers to reviewers,” Technical Report MS-CIS-08-30, University of Pennsylvania, 2008.
J. J. Merelo-Guervós, and P. Castillo-Valdivieso,” Conference paper assignment using a combined greedy/evolutionary algorithm,” Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. 3242, pp. 602-611, 2004. https://doi.org/10.1007/978-3-540-30217-9_61 .
S. L. Janak, M. S., Taylor, C. A. Floudas, M. K. Burka, and T. J. Mountziaris, “A novel and effective integer optimization approach for the Nsf panel assignment problem: A multi-resource and preference-constrained generalized assignment problem,” in Proceedings of AIChE Annual Meeting, no. 609, pp. 6066, 2005.
N. N. Li, Z. Zhao, J. H. Gu, and B. Y. Liu,. “Ant colony optimization algorithm for expert assignment problem,” in Proceedings of the 7th International Conference on Machine Learning and CyberneticsI 2008, July, pp. 660-664. https://doi.org/10.1109/ICMLC.2008.4620487 .
D. Conry, Y. Koren, and N. Ramakrishnan, “Recommender systems for the conference paper assignment problem,” in Proceedings of the 3rd ACM Conference on Recommender Systems, 2009, pp. 357-360. https://doi.org/10.1145/1639714.1639787.
B. Kat, “An algorithm and a decision support system for the panelist assignment problem: The case of TUBITAK,” Journal of the Faculty of Engineering and Architecture of Gazi University, no. 36 (1), pp. 69-87, 2021. https://doi.org/10.17341/gazimmfd.631071 .
F. Wang, S. Zhou, and N. Shi, “Group-to-group reviewer assignment problem,” Computers and Operations Research, no. 40 (5), pp. 1351-1362, 2013. https://doi.org/10.1016/j.cor.2012.08.005 .
D. K. Pradhan, J. Chakraborty, P. Choudhary, and S. Nandi, “An automated conflict of interest based greedy approach for conference paper assignment system,” Journal of Informetrics, no. 14 (2), 2020. https://doi.org/10.1016/j.joi.2020.101022 .
Y. Xu, J. Ma, Y. Sun, G. Hao, W. Xu, and D. Zhao, “A decision support approach for assigning reviewers to proposals,” Expert Systems with Applications, no. 37 (10), pp. 6948-6956, 2010. https://doi.org/10.1016/j.eswa.2010.03.027 .
N. N. Li, J. N. Zhang, J. H. Gu, and B. Y. Liu, “Solving expert assignment problem using improved genetic algorithm,” in Proceedings of the Sixth International Conference on Machine Learning and Cybernetics, 2007, 2 August, pp. 934-937. https://doi.org/10.1109/ICMLC.2007.4370276 .
J. Li, J. Peng, and Y. Wei, Adaptive parallel genetic algorithm for expert assignment problem. In Proceedings of 6th International Symposium on Computational Intelligence and Design, no. 1, pp. 23-26, 2013. https://doi.org/10.1109/ISCID.2013.13 .
C. Yang, T. Liu, W. Yi, X. Chen, and B. Niu, “Identifying expertise through semantic modeling: A modified BBPSO algorithm for the reviewer assignment problem,” Applied Soft Computing Journal, no. 94, pp. 106483, 2020. https://doi.org/10.1016/j.asoc.2020.106483 .
M. Huang, B. Liu, and L. Hong, “On assigning papers to reviewers,” 2nd International Workshop on Database Technology and Applications, 2010. https://doi.org/10.1109/DBTA.2010.5659009 .
Downloads
-
PDF (Українська)
Downloads: 113
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
