This issue was addressed by ad-justing the class weights in order to stronger penalize false positives when training the random forest model.
the class weight of the negative class was set to be num non replies num replies while the class of the positive class was 1.