Node by example

Node by example: 9. multipart module

9. multipart module

The complete source code can be downloaded here: http://github.com/Hendrik/node-by-example

Please note: the multipart module was originally part of node, but has been removed in a previous release.
It is supposed to come back in an improved form, so that I will keep this chapter separate from the "Addon" chapter.

First, in order to follow the example the old multipart module is needed. It is already included in the chapter 9 source code: 09_fileupload/multipart_old.js
This file was provided by: http://github.com/isaacs/multipart-js/raw/v0.0.0/lib/old.js

This multipart example is a little bit longer than previous code samples, so that I am going to provide just the line-by-line explanation. You can download the full code here: http://github.com/Hendrik/node-by-example

What does this example do?
Basically it provides a http file upload form, where the file gets streamed as a multipart.Stream and saved to an "upload" directory.
This example is based on a post on the debuggable.com site: http://debuggable.com/posts/streaming-file-uploads-with-node-js:4ac094b2-b6c8-4a7f-bd07-28accbdd56cb

First we need to include all the required modules.
fs is needed for the file functions, url for the url parsing, http to provide the http server and multipart_old to process the multipart.Stream:

var   sys = require("sys"),
       fs = require('fs'),
      url = require('url'),
     http = require("http"),
multipart = require("./multipart_old");

 

Next we define some variables used to save the (form field) name, filename and file (stream object):

var name, filename, file;

 

The HTTP server is basically identical to the example from the previous chapter with the exception of an added "/upload" page, which will call the upload_file() function:

http.createServer(function (request, response) {
  switch (url.parse(request.url).pathname) {
    case '/':
      display_form(request, response);
      break;
    case '/upload':
      upload_file(request, response);
      break;
    default:
      show_404(request, response);
      break;
  }
}).listen(8000);

 

First we start with the upload form page:

function display_form(request, response) {
  response.writeHead(200, {'Content-Type': 'text/html'});
  response.write(
    '
' + '' + '' + '
' ); response.end(); }

 

If you upload a file through the form, it will route you to the "/upload" page, where the juicy part begins.
First we set the encoding to binary, since a multipart element is being uploaded:

function upload_file(request, response) {
  request.setBodyEncoding('binary');
  ...
}

 

Then we create the multipart.Stream object:

var stream = new multipart.Stream(request);

 

This stream object emits the following events:
partBegin: emits once for every multipart "part" in the stream
body: emits when data is received; emits several times, since the data is streamed to the server
partEnd: emits once for every multipart "part" in the stream
complete: emits once the stream completed

Lets start with the "partBegin" event:
Here we save the current uploaded file's details and stream in the name, filename and file variables.
The "part" object contains all the submitted data from the client; feel free to uncomment the sys.debug() line to display the complete "part" details.

stream.addListener('partBegin', function(part) {
//  sys.debug(sys.inspect(part));
  name = part.name;
  filename = part.filename;
  file = fs.createWriteStream("./upload/" + filename);
});

 

Every time data is received through the stream the "body" event will emit, so we need to keep writing the data to the file with every emit:

stream.addListener('body', function(chunk) {
  file.write(chunk, function(err, bytesWritten) {
    sys.debug('bytes written: ' + bytesWritten);
  });
});

 

Once the stream has ended we can close the file stream:

stream.addListener('partEnd', function(part) {
  file.end();
});

 

There's nothing to do for the "complete" event, so just return once the stream is complete:

stream.addListener('complete', function() {
  return;
});

 

Once the file has been uploaded and saved in the upload directory, display a simple 'thanks' message to the client:

response.writeHead(200, {'Content-Type': 'text/plain'});
response.write('Thanks for the upload');
response.end();

 

...and, as seen in the previous chapter, the 404 page:

function show_404(request, response) {
  response.writeHead(404, {'Content-Type': 'text/plain'});
  response.write('404 - Please try again.');
  response.end();
}

 

That's all.
Make sure the "upload" directory is writable and then start the example code to test out the file upload.

One thing I was experimenting with was displaying a progress on the current upload. However, unfortunately there is no 100% correct way to get the exact file size before it has been saved on the server.
It is possible to calculate a simple estimated value, but I still found it ending up way too high or too low, depending on the submitted file size, so that the progress was more of a lucky guess than a correct information.
Maybe this will be available in the upcoming re-release of the multipart addon.

Comments

Hendrik wrote on July 23, 2010:
Hi Thomas, did you give it a try with various filesizes? I tried a similar approach, but would always end up with somewhat accurate results for very large files or for very small ones but not with a universal solution that would cover both. Especially with the smaller files it often came up with a 100% or higher difference.


Thomas wrote on July 25, 2010:
I did, only I didn't know what I know now, when writing file chunks the encoding should be set to binary: file.write(chunk, 'binary', function(err, bytesWritten) {}); I have tested it with various files and it is very accurate now. The only problem I am still facing now is how to stream this data to the client. Ajax polling doesn't seem a effecient solution so I guess I'll have to wait for full WebSocket support in all browsers. Here's the (modified) code to display upload progress in node js: http://pastebin.com/Mg5XjaGS


Thomas wrote on July 25, 2010:
Oops, I thought I missed something, this one's better: http://pastebin.com/CK35gGc0 (Works with the latest multipart.js + some other improvements


Thomas wrote on July 23, 2010:
Hello Hendrik, Your examples are very useful, thank you for them! I experimented with the upload progress thing too, and I found a way which works nearly perfect. The content-length header is almost equal to the actual file size, in fact it is a bit larger (only a few bytes). So first I stored this value in a variable size: size = part.parent.headers['content-length']; (In partBegin listener). Then I used the body listener to update the variable 'received': received += bytesWritten; Very simple so far. I used this code to display the upload progress: sys.debug('progress: ' + Math.round(received / size * 100)); Unfortunaly, it ended at 150% instead of 100. Somehow the total bytes that are written are 1/3 greater than the file size. So I made the following change: sys.debug('progress: ' + Math.round(received / size * 66.7)); And now it works perfectly, I tested it by comparing it against google chrome's statusbar (shows also progress on uploading). Greetings, Thomas