As for data science using Python, something tells me that this has to do with memory heap capacities. I’m not sure about Python’s max memory heap, but Javascript through Node.js seems to have only 512MB. I’ve been using Node.js to deal with big datasets and my most recent experimentation stumbled across the need of loading 100 million numbers to the RAM: while my PC has a fair amount of physical RAM (12GB) and a great part of it was available, it’ll simply error when filling an array. I needed an additional parameter, --max-old-space-size, so Node.js could deal with such amount of data. I didn’t try the same task with Python because I’m used to Javascript (yet I’m done some things in Python), but I wonder how much memory can Python hold until an error like “out of memory” happens, because ML models (for example, those hosted and served in HuggingFace) loads training weights with dozens of GBs
As for data science using Python, something tells me that this has to do with memory heap capacities. I’m not sure about Python’s max memory heap, but Javascript through Node.js seems to have only 512MB. I’ve been using Node.js to deal with big datasets and my most recent experimentation stumbled across the need of loading 100 million numbers to the RAM: while my PC has a fair amount of physical RAM (12GB) and a great part of it was available, it’ll simply error when filling an array. I needed an additional parameter,
--max-old-space-size, so Node.js could deal with such amount of data. I didn’t try the same task with Python because I’m used to Javascript (yet I’m done some things in Python), but I wonder how much memory can Python hold until an error like “out of memory” happens, because ML models (for example, those hosted and served in HuggingFace) loads training weights with dozens of GBs