Change user territory rights

Change user territory rights

By mistake I assigned a new user All Territories and then his own (GCC) territory.  I want to take back that permission but cannot; I have de-cativated him and when I re-activate he comes back with the same permission (all territories).

He (and others) are local sales offices and not actual employees and so cannot be allowed access to other accounts, etc.