These aren't on most color wheels, but they're considered neutral because they don't contrast with much of anything.
Neutral Color Scheme Wheel / On contemporary color wheels, neutrals are usually found at the far center or periphery of the types of neutral colors.