Is there a way to hide forum stats by Permission Sets as opposed to commenting them out?

As an example, I would want the stats panel hidden from Guests but have them viewable to Members. Permission sets seems like an ideal way to manage these kind of permissions.

Thanks
DD