Configuration option for treating integers as colors

For my work, I copy and paste a LOT of 4- to 6-digit numbers which Paste is considering Hex colors. I'd love an option to either stop treating these values as colors OR require the octothorpe (#) prefix be present to be identified as a color. Thanks!