Either have an admin controlling the kick or have a vote between all the users. With input for kicking unwanted users or from accidentally adding users.
