There are a lot of possible reasons for this, one of the first being that PandoraHearts has a pretty slow start compared to other manga. Until you get to volume five (or in some cases later), you might not find PandoraHearts interesting at all. Even then the story line is (arguably) somewhat cliche, and when things really start is probably when Elliot dies and everything starts going crazy and getting about a hundred times more complicated.
For those who do keep reading PandoraHearts, you might find the story getting too complicated or psychological for you to properly understand it. Though the fandom adores PandoraHearts for its complexity and sadness, some stop reading PandoraHearts for those very things, liking better the lighthearted manga it used to be. (If you delve into the mentally messed-up characters you'll realize things were never "lighthearted," but you would never know that unless you got farther into the manga and thought about it more than the manga provided.)
In rare cases, I've heard instances of people getting scared by the gore or quitting PandoraHearts after their favorite character was killed off. I actually heard about some boys refusing to read PandoraHearts simply because the title sounded too girly for their taste. (That's pretty silly, if you ask me.)
The last reason is that PandoraHearts just isn't known by a lot of people for several different reasons. Black Butler came out a tiny bit later than PandoraHearts did, and it received a much better anime than PandoraHearts did and thus became extremely popular. (A lot of Black Butler fans refuse to read PandoraHearts because they wrongly think it's a rip-off of Black Butler itself, not realizing PandoraHearts actually came out before Black Butler did. Also, as you know, the stories differ greatly, but one might not notice this if they only read the first, say, three volumes.) The anime we received for PandoraHearts may have had an amazing soundtrack, but the animation was awful and the way they portrayed the story just didn't work, not to mention it was canceled very early in.