Outside of the immunity to Charm it was still a decent use of an action/ready action.
I think you could rule either way on if the zombies/lich would jump off a cliff after the gem, since the spell mentions that they specifically stare at and approach you (not the gem).
So you better jump off and cast feather fall.
I’ve heard it both ways.