Fake news 3: How do you build a fake news prediction model, what features will you use, and evaluate the model performance.
Instruction: Define the target labels, evidence, model baseline, features, evaluation split, and error measures; explain how uncertain claims and harmful false positives would be handled.
Updated
Example Answer
I would first define the labels and the evidence required for them. A model trained on "fake news" labels can otherwise learn a publisher or writing style instead of checking the claim. I would include an uncertain outcome and review disagreements rather than force every item into true or false.
For a baseline, I would compare a simple text classifier with an approach that uses retrieved evidence. Features could include the claim text and its relationship to the evidence, while source or style features would need careful testing for shortcuts. I would keep versions of the same story in one split and test on later material or unseen publishers. I would report precision and recall for the defined target class, coverage when the system abstains, and important error examples. F1 alone would not justify automated labeling or removal.
Make it your own
Name the target label and how reviewers substantiate it. Explain the cost of falsely labeling accurate reporting and how uncertain items are handled. Evaluate the actual deployment population rather than relying on a random split of duplicated articles.
Why this works
It covers labels, features, leakage-resistant evaluation, and the consequences of errors.
Follow-up to practice
How would you detect a model that is mostly recognizing publishers?