Publications

2024

2024

1. Preprint

   WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

   Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Xiaofeng Xu, Xia Song, and Jennifer Neville

   2024

   Abs arXiv Bib HTML

   As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages real-time, in-situ user interactions to create preference datasets that more accurately reflect authentic human values. WildFeedback operates through a three-step process: feedback signal identification, preference data construction, and user-guided evaluation. We applied this framework to a large corpus of user-LLM conversations, resulting in a rich preference dataset that reflects genuine user preferences. This dataset captures the nuances of user preferences by identifying and classifying feedback signals within natural conversations, thereby enabling the construction of more representative and context-sensitive alignment data. Our extensive experiments demonstrate that LLMs fine-tuned on WildFeedback exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed user-guided evaluation. By incorporating real-time feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users. In summary, WildFeedback offers a robust, scalable solution for aligning LLMs with true human values, setting a new standard for the development and evaluation of user-centric language models.

   @misc{shi2024wildfeedbackaligningllmsinsitu,
     title = {WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback},
     author = {Shi, Taiwei and Wang, Zhuoer and Yang, Longqi and Lin, Ying-Chun and He, Zexue and Wan, Mengting and Zhou, Pei and Jauhar, Sujay and Xu, Xiaofeng and Song, Xia and Neville, Jennifer},
     year = {2024},
     eprint = {2408.15549},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     url = {https://arxiv.org/abs/2408.15549},
   }

2. Preprint

   DHP Benchmark: Are LLMs Good NLG Evaluators?

   Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, Mark Cusick, Param Kulkarni, Zhengping Ji, Yasser Ibrahim, and Xia Hu

   2024

   Abs arXiv Bib

   Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks. However, the capabilities of LLMs in scoring NLG quality remain inadequately explored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs utilizing hierarchically perturbed text data and statistical tests to measure the NLG evaluation capabilities of LLMs systematically. We have re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM series provides critical insight into their strengths and limitations as NLG evaluators.

   @misc{wang2024dhpbenchmarkllmsgood,
     title = {DHP Benchmark: Are LLMs Good NLG Evaluators?},
     author = {Wang, Yicheng and Yuan, Jiayi and Chuang, Yu-Neng and Wang, Zhuoer and Liu, Yingchi and Cusick, Mark and Kulkarni, Param and Ji, Zhengping and Ibrahim, Yasser and Hu, Xia},
     year = {2024},
     eprint = {2408.13704},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     url = {https://arxiv.org/abs/2408.13704},
   }

3. EMNLP
   Findings

   FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking

   Zhuoer Wang, Leonardo F. R. Ribeiro, Alexandros Papangelis, Rohan Mukherjee, Tzu-Yen Wang, Xinyan Zhao, Arijit Biswas, James Caverlee, and Angeliki Metallinou

   In Findings of the Association for Computational Linguistics: EMNLP 2024, Nov 2024

   Abs arXiv Bib PDF Slides

   API call generation is the cornerstone of large language models’ tool-using ability that provides access to the larger world. However, existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user’s request. To address these limitations, we propose an output-side optimization approach called FANTASE. Two of the unique contributions of FANTASE are its State-Tracked Constrained Decoding (SCD) and Reranking components. SCD dynamically incorporates appropriate API constraints in the form of Token Search Trie for efficient and guaranteed generation faithfulness with respect to the API documentation. The Reranking component efficiently brings in the supervised signal by leveraging a lightweight model as the discriminator to rerank the beam-searched candidate generations of the large language model. We demonstrate the superior performance of FANTASE in API call generation accuracy, inference efficiency, and context efficiency with DSTC8 and API Bank datasets.

   @inproceedings{wang-etal-2024-fantase,
     title = {FANTAstic SEquences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking},
     author = {Wang, Zhuoer and Ribeiro, Leonardo F. R. and Papangelis, Alexandros and Mukherjee, Rohan and Wang, Tzu-Yen and Zhao, Xinyan and Biswas, Arijit and Caverlee, James and Metallinou, Angeliki},
     year = {2024},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
     month = nov,
     address = {Miami, United States},
     publisher = {Association for Computational Linguistics},
     eprint = {2407.13945},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     url = {https://arxiv.org/abs/2407.13945},
   }

2023

2023

1. EMNLP
   Findings

   Unsupervised Candidate Answer Extraction through Differentiable Masker-Reconstructor Model

   Zhuoer Wang, Yicheng Wang, Ziwei Zhu, and James Caverlee

   In Findings of the Association for Computational Linguistics: EMNLP 2023, Dec 2023

   Abs arXiv Bib PDF Code Poster

   Question generation is a widely used data augmentation approach with extensive applications, and extracting qualified candidate answers from context passages is a critical step for most question generation systems. However, existing methods for candidate answer extraction are reliant on linguistic rules or annotated data that face the partial annotation issue and challenges in generalization. To overcome these limitations, we propose a novel unsupervised candidate answer extraction approach that leverages the inherent structure of context passages through a Differentiable Masker-Reconstructor (DMR) Model with the enforcement of self-consistency for picking up salient information tokens. We curated two datasets with exhaustively-annotated answers and benchmark a comprehensive set of supervised and unsupervised candidate answer extraction methods. We demonstrate the effectiveness of the DMR model by showing its performance is superior among unsupervised methods and comparable to supervised methods.

   @inproceedings{wang-etal-2023-dmr,
     title = {Unsupervised Candidate Answer Extraction through Differentiable Masker-Reconstructor Model},
     author = {Wang, Zhuoer and Wang, Yicheng and Zhu, Ziwei and Caverlee, James},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
     month = dec,
     year = {2023},
     address = {Singapore},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.findings-emnlp.379/},
   }

2. EMNLP
   Findings

   Co^2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning

   Xiangjue Dong, Ziwei Zhu, Zhuoer Wang, Maria Teleki, and James Caverlee

   In Findings of the Association for Computational Linguistics: EMNLP 2023, Dec 2023

   Abs arXiv Bib PDF Code Poster

   Pre-trained Language Models are widely used in many important real-world applications. However, recent studies show that these models can encode social biases from large pre-training corpora and even amplify biases in downstream applications. To address this challenge, we propose Co2PT, an efficient and effective debias-while-prompt tuning method for mitigating biases via counterfactual contrastive prompt tuning on downstream tasks. Our experiments conducted on three extrinsic bias benchmarks demonstrate the effectiveness of Co2PT on bias mitigation during the prompt tuning process and its adaptability to existing upstream debiased language models. These findings indicate the strength of Co2PT and provide promising avenues for further enhancement in bias mitigation on downstream tasks.

   @inproceedings{dong-etal-2023-co2pt,
     title = {Co^2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning},
     author = {Dong, Xiangjue and Zhu, Ziwei and Wang, Zhuoer and Teleki, Maria and Caverlee, James},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
     month = dec,
     year = {2023},
     address = {Singapore},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.findings-emnlp.390/},
   }

3. ACL
   Outstanding Paper Award

   Faithful Low-Resource Data-to-Text Generation through Cycle Training

   Zhuoer Wang, Marcus Collins, Nikhita Vedula, Simone Filice, Shervin Malmasi, and Oleg Rokhlenko

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023

   Abs arXiv Bib PDF Code Poster Slides

   Methods to generate text from structured data have advanced significantly in recent years, primarily due to fine-tuning of pre-trained language models on large datasets. However, such models can fail to produce output faithful to the input data, particularly on out-of-domain data. Sufficient annotated data is often not available for specific domains, leading us to seek an unsupervised approach to improve the faithfulness of output text. Since the problem is fundamentally one of consistency between the representations of the structured data and text, we evaluate the effectiveness of cycle training in this work. Cycle training uses two models which are inverses of each other: one that generates text from structured data, and one which generates the structured data from natural language text. We show that cycle training, when initialized with a small amount of supervised data (100 samples in our case), achieves nearly the same performance as fully supervised approaches for the data-to-text generation task on the WebNLG, E2E, WTQ, and WSQL datasets. We perform extensive empirical analysis with automated evaluation metrics and a newly designed human evaluation schema to reveal different cycle training strategies’ effectiveness of reducing various types of generation errors.Our code is publicly available at https://github.com/Edillower/CycleNLG.

   @inproceedings{wang-etal-2023-faithful,
     title = {Faithful Low-Resource Data-to-Text Generation through Cycle Training},
     author = {Wang, Zhuoer and Collins, Marcus and Vedula, Nikhita and Filice, Simone and Malmasi, Shervin and Rokhlenko, Oleg},
     booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = jul,
     year = {2023},
     address = {Toronto, Canada},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.acl-long.160},
     pages = {2847--2867},
   }

2022

2022

1. ALEXA PRIZE

   Howdy Y’all: An Alexa TaskBot

   Texas A&M University

   In Alexa Prize TaskBot Challenge Proceedings, Jul 2022

   Abs Bib HTML PDF

   In this paper, we present Howdy Y’all, a multi-modal task-oriented dialogue agent developed for the 2021-2022 Alexa Prize TaskBot competition. Our design principles guiding Howdy Y’all aim for high user satisfaction through friendly and trustworthy encounters, minimization of negative conversation edge cases, and wide coverage over many tasks. Hence, Howdy Y’all is built upon a rapid prototyping platform to enable fast experimentation and powered by four key innovations to enable this vision: (i) First, it combines a rules, phonetic matching, and a transformer-based approach for robust intent understanding. (ii) Second, to accurately elicit user preferences and guide users to the right task, Howdy Y’all is powered by a contrastive learning search framework over sentence embeddings and a conversational recommender for eliciting preferences. (iii) Third, to support a variety of user question types, it introduces a new data augmentation method for question generation and a self-supervised answer selection approach for improving question answering. (iv) Finally, to help motivate our users and keep them engaged, we design an emotional conversation tracker that provides empathetic responses to keep users engaged and a monitor of conversation quality.

   @inproceedings{howdyyall-2022,
     author = {University, Texas A&M},
     title = {Howdy Y’all: An Alexa TaskBot},
     year = {2022},
     booktitle = {Alexa Prize TaskBot Challenge Proceedings},
   }

2. AAAI
   Demo

   RES: An Interpretable Replicability Estimation System for Research Publications

   Zhuoer Wang, Qizhang Feng, Mohinish Chatterjee, Xing Zhao, Yezi Liu, Yuening Li, Abhay Kumar Singh, Frank M. Shipman, Xia Hu, and James Caverlee

   In Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2022

   Abs Bib HTML PDF Poster

   Reliable and faithful research is the cornerstone of breakthrough advancements and disruptive innovations. Assessing the credibility of scientific findings and claims in research publications has long been a time-consuming and challenging task for researchers and decision-makers. In this paper, we introduce RES - an intelligent system that assists humans in analyzing the credibility of scientific findings and claims in research publications in the field of social and behavioral sciences by estimating their replicability. The pipeline of RES consists of four major modules that perform feature extraction, replicability estimation, result explanation, and sentiment analysis respectively. Our evaluation based on human experts’ assessments suggests that the RES has achieved adequate performance. The RES is also built with a Graphical User Interface (GUI) that is publicly accessible at https://tamu-infolab.github.io/RES

   @inproceedings{wang-etal-2022-res,
     title = {RES: An Interpretable Replicability Estimation System for Research Publications},
     author = {Wang, Zhuoer and Feng, Qizhang and Chatterjee, Mohinish and Zhao, Xing and Liu, Yezi and Li, Yuening and Singh, Abhay Kumar and Shipman, Frank M. and Hu, Xia and Caverlee, James},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
     year = {2022},
     month = jun,
     volume = {36},
     number = {11},
     pages = {13230-13232},
   }

2021

2021

1. arXiv

   Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models

   Jian Wu, Rajal Nivargi, Sree Sai Teja Lanka, Arjun Manoj Menon, Sai Ajay Modukuri, Nishanth Nakshatri, Xin Wei, Zhuoer Wang, James Caverlee, Sarah M Rajtmajer, and C. Lee Giles

   arXiv preprint arXiv:2104.04580, Jun 2021

   Abs arXiv Bib PDF

   In recent years, significant effort has been invested verifying the reproducibility and robustness of research claims in social and behavioral sciences (SBS), much of which has involved resource-intensive replication projects. In this paper, we investigate prediction of the reproducibility of SBS papers using machine learning methods based on a set of features. We propose a framework that extracts five types of features from scholarly work that can be used to support assessments of reproducibility of published research claims. Bibliometric features, venue features, and author features are collected from public APIs or extracted using open source machine learning libraries with customized parsers. Statistical features, such as p-values, are extracted by recognizing patterns in the body text. Semantic features, such as funding information, are obtained from public APIs or are extracted using natural language processing models. We analyze pairwise correlations between individual features and their importance for predicting a set of human-assessed ground truth labels. In doing so, we identify a subset of 9 top features that play relatively more important roles in predicting the reproducibility of SBS papers in our corpus. Results are verified by comparing performances of 10 supervised predictive classifiers trained on different sets of features.

   @article{wu2021predicting,
     title = {Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models},
     author = {Wu, Jian and Nivargi, Rajal and Lanka, Sree Sai Teja and Menon, Arjun Manoj and Modukuri, Sai Ajay and Nakshatri, Nishanth and Wei, Xin and Wang, Zhuoer and Caverlee, James and Rajtmajer, Sarah M and Giles, C. Lee},
     journal = {arXiv preprint arXiv:2104.04580},
     year = {2021},
   }

2020

2020

1. EMNLP

   PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge

   Yun He, Zhuoer Wang, Yin Zhang, Ruihong Huang, and James Caverlee

   In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov 2020

   Abs Bib PDF Code

   We present a new benchmark dataset called PARADE for paraphrase identification that requires specialized domain knowledge. PARADE contains paraphrases that overlap very little at the lexical and syntactic level but are semantically equivalent based on computer science domain knowledge, as well as non-paraphrases that overlap greatly at the lexical and syntactic level but are not semantically equivalent based on this domain knowledge. Experiments show that both state-of-the-art neural models and non-expert human annotators have poor performance on PARADE. For example, BERT after fine-tuning achieves an F1 score of 0.709, which is much lower than its performance on other paraphrase identification datasets. PARADE can serve as a resource for researchers interested in testing models that incorporate domain knowledge. We make our data and code freely available.

   @inproceedings{he-etal-2020-parade,
     title = {{PARADE}: {A} {N}ew {D}ataset for {P}araphrase {I}dentification {R}equiring {C}omputer {S}cience {D}omain {K}nowledge},
     author = {He, Yun and Wang, Zhuoer and Zhang, Yin and Huang, Ruihong and Caverlee, James},
     booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
     month = nov,
     year = {2020},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.emnlp-main.611},
     doi = {10.18653/v1/2020.emnlp-main.611},
     pages = {7572--7582},
   }

2018

2018

1. NT

   Fault propagation and effects analysis for designing an online monitoring system for the secondary loop of the nuclear power plant portion of a hybrid energy system

   Xiaoxu Diao, Yunfei Zhao, Mike Pietrykowski, Zhuoer Wang, Shannon Bragg-Sitton, and Carol Smidts

   Nuclear Technology, Nov 2018

   Abs Bib HTML

   This paper studies the propagation and effects of faults in critical components that pertain to the secondary loop of a nuclear power plant found in nuclear hybrid energy systems (NHESs). This information is used to design an online monitoring (OLM) system that is capable of detecting and analyzing faults that are likely to occur during NHES operation. In this research, the causes, features, and effects of possible faults are investigated by simulating the propagation of faults in the secondary loop of a nuclear power plant. The simulation is conducted using Integrated System Failure Analysis (ISFA), a promising method analyzing hardware and software faults during the conceptual design phase. First, the models of system components required by ISFA are constructed. Then, fault propagation analysis is implemented, which is conducted under the bounds set by acceptance criteria derived for the design of an OLM system. The result of the fault simulation is utilized to evaluate the effectiveness of signals for fault detection and diagnosis and to propose an optimization plan for the OLM system. Finally, several experiments are designed and conducted using a hardware-in-the-loop system to verify the correctness and effectiveness of the proposed method.

   @article{diao2018fault,
     title = {Fault propagation and effects analysis for designing an online monitoring system for the secondary loop of the nuclear power plant portion of a hybrid energy system},
     author = {Diao, Xiaoxu and Zhao, Yunfei and Pietrykowski, Mike and Wang, Zhuoer and Bragg-Sitton, Shannon and Smidts, Carol},
     journal = {Nuclear Technology},
     volume = {202},
     number = {2-3},
     pages = {106--123},
     year = {2018},
     publisher = {Taylor \& Francis},
   }