How would you evaluate replacing flat comments with a threaded comment structure?

Example answer

I would test whether threaded replies make conversations easier to follow and useful contributions easier to find. More comments, deeper threads or longer reading time can also reflect arguments or navigation difficulty.

Define a primary outcome such as the proportion of sampled discussion visits in which users report finding a useful answer, with a consistent survey and response-rate reporting. Supporting measures could include successful reply-to-intended-comment actions, unanswered questions and sampled conversation-quality ratings. Keep moderation workload, abuse reports, accidental replies, loading time and accessibility as guardrails.

Compare the flat and threaded designs in an experiment that avoids showing inconsistent conversation structures within the same discussion. Assign whole threads or suitable communities where feasible and analyze at that clustered assignment level; document spillovers from people visiting several communities. Keep content and ranking changes separate where possible.

Prespecify the evaluation window and meaningful improvement, verify exposure and logging, and examine newcomers as well as frequent contributors. Qualitative tasks can explain confusion that activity counts miss. I would ship when evidence supports easier, healthier discussion without material guardrail harm, not merely because thread depth rose.