It is possible we have over-complicated this to some extent but this has always been done after receiving user requests and feedback. And – coincidentally – I have been thinking of proposing a change as it happens. But…
What you are almost certainly seeing is that you have supplied a main custom icon but not icons for the other states – ie., when there are unread posts in the forum. There are actually 3 states a forum can be in listed on the create/edit forum admin form. (Nothing new, new and unread posts, locked. In addition there are a further 5 states that a topic can be in which also can have a custom icon). So if your forum is in any one of those states – i.e., there are topics with unread posts, and if there is no custom icon for that, it will revert to the default icon supplied with the forum theme.
So – could that be the issue?
For the record – I have, as I said, been pondering this recently. We could offer to use the base custom icon if exists for all states. The problem with that is that it loses important functionality and visual clues. I would be interested in your opinion in fact.