Support Forum

I'm trying to limit the permissions of my private forums so that users of certain groups can see certain forums but can't enter them. I have created a new Permission Set, cloned No Access, then only checked "Can view a list of forums only". This works great, except if the user clicks on the forum, they are brought to a forum page with no threads in it. Is there another way I should be going about this so it doesn't result in a "broken forum" look? (concerned it won't be clear to my users what exactly is going on)