It is not realistic to make a direct comparison between the forum loading time and other pages within a WordPress site. WordPress posts and pages are static. The do not change from moment to moment, they can generally be shown to anyone (even if needing to be logged in to the site) and the data underpinning the display is very lightweight.
Forums are completely the opposite. On a busy forum, the data can be changing and being added to second my second – often with many users adding content simultaneously. This makes for a very rich environment but the cost will always be the extra ‘work’ the code and database has to do to load up a view. Remember too that the forum display is uniquely tailored for the current user loading the page. One user will see a different view than the next user depending on what the have and have not read, what their forum permissions are and what forum or topic they may be looking at. And even on a quiet forum, the checks and changes are still required.
I would suggest taking a look in our codex at this page however: https://simple-press.com/documentation/codex/installation/installation-information/performance-notes/