javascript - Is there a limit on the size of a string in JSON with Node.js?

Question

A section of my Node.js application involves receiving a string as input from the user and storing it in a JSON file. JSON itself obviously has no limit on this, but is there any upper bound on the amount of text that Node can process into JSON?

Note that I am not using MongoDB or any other technology for the actual insertion - this is native stringification and saving to a .json file using fs.

Answer 1

V8 (the JavaScript engine node is built upon) until very recently had a hard limit on heap size of about 1.9 GB.

Node v0.10 is stuck on an older version of V8 (3.14) due to breaking V8 API changes around native addons. Node 0.12 will update to the newest V8 (3.26), which will break many native modules, but opens the door for the 1.9 GB heap limit to be raised.

So as it stands, a single node process can keep no more than 1.9 GB of JavaScript code, objects, strings, etc combined. That means the maximum length of a string is under 1.9 GB.

You can get around this by using Buffers, which store data outside of the V8 heap (but still in your process's heap). A 64-bit build of node can pretty much fill all your RAM as long as you never have more than 1.9 GB of data in JavaScript variables.

---

All that said, you should never come anywhere near this limit. When dealing with this much data, you must deal with it as a stream. You should never have more than a few megabytes (at most) in memory at one time. The good news is node is especially well-suited to dealing with streaming data.

You should ask yourself some questions:

• What kind of data are you actually receiving from the user?
• Why do you want to store it in JSON format?
• Is it really a good idea to stuff gigabytes into JSON? (The answer is no.)
• What will happen with the data later, after it is stored? Will your code read it? Something else?

The question you've posted is actually quite vague in regard to what you're actually trying to accomplish. For more specific advice, update your question with more information.

If you expect the data to never be all that big, just throw a reasonable limit of 10 MB or something on the input, buffer it all, and use JSON.stringify.

If you expect to deal with data any larger, you need to stream the input straight to disk. Look in to transform streams if you need to process/modify the data before it goes to disk. For example, there are modules that deal with streaming JSON.