Can only group by visible fields?

Can only group by visible fields?

When using the "Set Grouping" in view configuration, it only lets you select fields to group by that are marked as visible in the column setup. Why is that? I can filter using fields that are not visible. Generally, in fact, you often wouldn't need to display a field that you are using to group the records ... since you are grouping by it! Please advise how I can group by a particular field in a view, but not display that column.
Thanks,
Craig