Issue with colours on my Ring Chart

Issue with colours on my Ring Chart

I have a ring chart with colours assigned to them with specific meanings: red, green, blue and grey. 
When one of them is missing the colours don't align eg: red becomes grey, grey becomes green.

Is there a way to make sure that the colours stay where they are supposed to?