Folders

I try to press 'x' to delete a folder, but it stays! How do I delete folders I don't want?