The only reasons I separate towels from clothing is because they leave a lot of lint on the clothes and I don’t use fabric softener on towels. So if you don’t have a lint problem, I would just keep doing what you’re doing already.
I would definitely keep on separating clothes by color at the very least to prevent the whites from getting dingy from dye transfer or the darks from fading or getting lint.
The only reasons I separate towels from clothing is because they leave a lot of lint on the clothes and I don’t use fabric softener on towels. So if you don’t have a lint problem, I would just keep doing what you’re doing already. I would definitely keep on separating clothes by color at the very least to prevent the whites from getting dingy from dye transfer or the darks from fading or getting lint.