There's a few reasons for this, but I'm fairly-certain the notoriety came about in Bram Stoker's Dracula.
Jonathan Harker enters the Count's castle as a guest and finds it lacking in mirrors. Harker brought his own mirror to shave with, but accidentally cuts himself in the process when the Count interrupts him. The Count becomes enticed by the freshly-drawn blood and lunges himself at Harker's throat, but is repelled by the rosary around his neck. He then lectures Harker about how unwise it is to cut himself in "this country" and throws the mirror out the window, but not before Harker takes notice that the Count bears no reflection.
The reasons for this vary from hearsay to hearsay.
- Dracula is soulless. (This suggests that every inanimate object found within a mirror's reflection has a soul. Beware of that cabinet, it seems "crafty.")
- As a "dark entity," Dracula is harmed by light/sunlight and, in turn, cannot reflect it.
- Dracula is an apparition of illusion/shapeshifting (capable of manifesting himself as a bat, wolf or mist), and bears no reflection simply because he chooses not to.
As a rule, Dracula is a monster, and monsters are as elusive as they are dangerous. Some stories even claim that Dracula cannot be seen by those he keeps his back to. Mirrors have always been a subject of superstition, though. (To this day, children still dare eachother into playing "Bloody Mary.") The same could be said of cameras—I imagine that a wide-variety of cultures felt superstitious upon introduction.
Perhaps the best evidence of this can be found in Japan during the 1800s, regarding Ueno Hikoma: revered photographer and son of Ueno Toshinojo (first to import a daguerreotype). Many Japanese attributed illness/death to the device, but Ueno eventually led a successful career and turned iconic following his death in the early 1900s. Some of his photographs can be found in the Freer Gallery of Art.