You can leave a channel, regardless of whether it is associated with a group. Leaving a channel will not remove you from the associated group. However, once you leave any Channel, only the admin(s) can add you back to it.