This morning I touched the metal handle of my office door and the wooden frame right beside it. Same room, same temperature reading on the wall—yet the metal felt noticeably colder. I nearly started explaining to a colleague that "the cold transfers faster from metal," before catching myself mid-sentence. That's the misconception talking.
There is no such thing as "cold" transferring. Cold isn't a substance or a force that flows between objects. It's the absence of heat, nothing more. What actually happened was that the metal conducted thermal energy away from my hand more efficiently than the wood did. My hand lost heat rapidly to the metal, creating that sharp cold sensation, while the wood—being a poor conductor—let me keep more of my body heat.
Think of it like darkness. We don't say darkness flows into a room when you turn off the lights. Darkness is simply what we call the absence of light. Cold works the same way: it's our name for low thermal energy, for the relative absence of heat. Heat is the thing that moves, always flowing from warmer objects to cooler ones until equilibrium is reached.