I always say - to defend the series (which doesn’t need too much defending, it’s the most successful book series after the old testament > new testament > Quran trilogy). The magic of Harry Potter is that all of the fantasy magic works exactly as well as it needs to right at the moment that it’s directly in front of the readers eyes. As you mention, as soon as it leaves the view of the characters in the story, it literally blows up into nonsense. However, as the story is being told the magic used is awesome and just what the plot needs at that exact moment to move along.