Cast on Minion Death Bugged on Weapon Swap

I have a Cast on Minion Death Gem. It works with the Weapon Set where my Sceptre is. But once I try to have it active on my other Weapon Set, it doesn't work. It tells me that I don't have a spell Socketed, which isn't true obviously because I have Comet in there and it works on the other Weapon Set. I tried taking out Upwhelling, putting the Comet in and out. Setting it to work with certain weapon sets only. Desummoning all my Minions. Nothing seams to work. I can only ever use it with one Weapon set no matter what.

It sucks really because there is Quality and a Perfect Jewelers Orb on there. It also seams like it doesn't properly lock my Sceptre to both hands even if I lock it. Maybe the bug is related to that.