Speed mostly comes down to resources – a good server – preferably non-shared – and enough PHP memory to cope easily. However – there are two questions always worth asking.
Firstly – are you using the SP Admin Bar plugin and if so – do you maintain and keep the admin’s Postbag empty. if you let that build up then it can slow things down noticeably for the admins and moderators.
Secondly – do you use Gravatars as avatars on your forums? If so then these can be a very big drag on performance and the best way to deal with that is to install the SP Gravatar Cache plugin.
Actually – I personally find this site is loading far too slowly for my comfort. Although – to be fair – that is mostly down to accessing Twitter (for the sidebar) and – I hate to say it – the ads server.