Can I restrict the Forum by user?

No.  There are no specific authorizations that can allow or restrict access to the Forum.  The Forum is either turned on or off at the organization level.  The only way to keep a specific user out of the Forum is to suspend their access to Schedule Master.