With our recent enhancement to custom button creation from the layout editor, administrators can now reorder custom buttons directly from the layout editor, and this order will be reflected for all users.
In addition to this, individual users can pin the custom buttons they use most frequently and rearrange the order of their pinned buttons based on their own preference.
You can find more details about this enhancement here: