Check with users you're removing to make sure they are no longer actually working on the workbook. If you remove an active user, the user's unsaved work will be lost.
If a user no longer needs to work with a shared workbook, you can reduce the size of the workbook file by deleting the user's personal view settings. On the View menu, click Custom Views. In the Views box, click the user's view, and then click Delete.