Benchmark for misrepresented social media responses to government posts

Government decisions are regularly announced via posts in several internet-based media such as news websites and social media. In response to government posts, users are able to add their comments and publish (e.g., retweet, repost) to such media. Social media responses by the community are not only indicators of public perception but may also be the starting point for misinformation. While social media responses may largely be innocuous, some may express displeasure via satire, sarcasm, mockery or, in some cases, hate speech. These may not necessarily be directed at the government post but may be an intentional or unintentional misrepresentation.  The proposed project will conduct a fine-grained evaluation of detecting misrepresentations in social media responses to government posts. With a government post (i.e., long text in the form of a news article or press release) and a social media response (i.e., a short post that quotes a government post; we will look at social media platforms such as Reddit and X) as input, we will evaluate the ability of large language models (LLMs) to detect misrepresentation across four dimensions: hate speech, sarcasm, violent threat and falsehood. The scope of the project is to create a benchmark for fine-grained misrepresentation detection. A benchmark consists of a manually labelled dataset along with performance report of a range of LLMs to automatically predict the labels.