Node by example

Node by example: 9. multipart module

9. multipart module

The complete source code can be downloaded here: http://github.com/Hendrik/node-by-example

Please note: the multipart module was originally part of node, but has been removed in a previous release.
It is supposed to come back in an improved form, so that I will keep this chapter separate from the "Addon" chapter.

First, in order to follow the example the old multipart module is needed. It is already included in the chapter 9 source code: 09_fileupload/multipart_old.js
This file was provided by: http://github.com/isaacs/multipart-js/raw/v0.0.0/lib/old.js

This multipart example is a little bit longer than previous code samples, so that I am going to provide just the line-by-line explanation. You can download the full code here: http://github.com/Hendrik/node-by-example

What does this example do?
Basically it provides a http file upload form, where the file gets streamed as a multipart.Stream and saved to an "upload" directory.
This example is based on a post on the debuggable.com site: http://debuggable.com/posts/streaming-file-uploads-with-node-js:4ac094b2-b6c8-4a7f-bd07-28accbdd56cb

First we need to include all the required modules.
fs is needed for the file functions, url for the url parsing, http to provide the http server and multipart_old to process the multipart.Stream:

var   sys = require("sys"),
       fs = require('fs'),
      url = require('url'),
     http = require("http"),
multipart = require("./multipart_old");

 

Next we define some variables used to save the (form field) name, filename and file (stream object):

var name, filename, file;

 

The HTTP server is basically identical to the example from the previous chapter with the exception of an added "/upload" page, which will call the upload_file() function:

http.createServer(function (request, response) {
  switch (url.parse(request.url).pathname) {
    case '/':
      display_form(request, response);
      break;
    case '/upload':
      upload_file(request, response);
      break;
    default:
      show_404(request, response);
      break;
  }
}).listen(8000);

 

First we start with the upload form page:

function display_form(request, response) {
  response.writeHead(200, {'Content-Type': 'text/html'});
  response.write(
    '
' + '' + '' + '
' ); response.end(); }

 

If you upload a file through the form, it will route you to the "/upload" page, where the juicy part begins.
First we set the encoding to binary, since a multipart element is being uploaded:

function upload_file(request, response) {
  request.setBodyEncoding('binary');
  ...
}

 

Then we create the multipart.Stream object:

var stream = new multipart.Stream(request);

 

This stream object emits the following events:
partBegin: emits once for every multipart "part" in the stream
body: emits when data is received; emits several times, since the data is streamed to the server
partEnd: emits once for every multipart "part" in the stream
complete: emits once the stream completed

Lets start with the "partBegin" event:
Here we save the current uploaded file's details and stream in the name, filename and file variables.
The "part" object contains all the submitted data from the client; feel free to uncomment the sys.debug() line to display the complete "part" details.

stream.addListener('partBegin', function(part) {
//  sys.debug(sys.inspect(part));
  name = part.name;
  filename = part.filename;
  file = fs.createWriteStream("./upload/" + filename);
});

 

Every time data is received through the stream the "body" event will emit, so we need to keep writing the data to the file with every emit:

stream.addListener('body', function(chunk) {
  file.write(chunk, function(err, bytesWritten) {
    sys.debug('bytes written: ' + bytesWritten);
  });
});

 

Once the stream has ended we can close the file stream:

stream.addListener('partEnd', function(part) {
  file.end();
});

 

There's nothing to do for the "complete" event, so just return once the stream is complete:

stream.addListener('complete', function() {
  return;
});

 

Once the file has been uploaded and saved in the upload directory, display a simple 'thanks' message to the client:

response.writeHead(200, {'Content-Type': 'text/plain'});
response.write('Thanks for the upload');
response.end();

 

...and, as seen in the previous chapter, the 404 page:

function show_404(request, response) {
  response.writeHead(404, {'Content-Type': 'text/plain'});
  response.write('404 - Please try again.');
  response.end();
}

 

That's all.
Make sure the "upload" directory is writable and then start the example code to test out the file upload.

One thing I was experimenting with was displaying a progress on the current upload. However, unfortunately there is no 100% correct way to get the exact file size before it has been saved on the server.
It is possible to calculate a simple estimated value, but I still found it ending up way too high or too low, depending on the submitted file size, so that the progress was more of a lucky guess than a correct information.
Maybe this will be available in the upcoming re-release of the multipart addon.

Node by example

Node by example: 8. HTTP module

8. HTTP module

The complete source code can be downloaded here: http://github.com/Hendrik/node-by-example

Node provides a module specifically for the http protocol. Use it via require('http').

As taken from the official documentation at http://nodejs.org/api.html#http-145:
The HTTP interfaces in Node are designed to support many features of the protocol which have been traditionally difficult to use. In particular, large, possibly chunk-encoded, messages. The interface is careful to never buffer entire requests or responses, the user is able to stream data.

In order to support the full spectrum of possible HTTP applications, Node's HTTP API is very low-level. It deals with stream handling and message parsing only. It parses a message into headers and body but it does not parse the actual headers or the body.

A simple HTTP server & client example, where the server provides an index page with argument support and a 404 page for any other requests.

Let's start with the server:
See 08_http/simple_server.js:

var     sys = require("sys"),
       http = require("http"),
        url = require("url"),
querystring = require("querystring");

http.createServer(function (request, response) {
  switch (url.parse(request.url).pathname) {
    case '/':
      show_index(request, response);
      break;
    default:
      show_404(request, response);
      break;
  }
}).listen(8000);

function show_index(request, response) {
  sys.puts("Serving index page");
  response.writeHead(200, {'Content-Type': 'text/html'});
  var output = 'node.js HTTP server exampleIndex output';
  var url_request = url.parse(request.url).query;
  output += "

Request query: " + url_request + "

"; if (url_request) sys.puts("Request query: " + JSON.stringify(querystring.parse(url_request))); output += ''; response.write(output); response.end(); } function show_404(request, response) { sys.puts("Serving 404 error page"); response.writeHead(404, {'Content-Type': 'text/plain'}); response.write('404 - Please try again.'); response.end(); }

 

The server will parse the requested URL for the requested pathname and its query string, which can be done by using the "url" and "querystring" modules (more on those later):

url = require("url"),
querystring = require("querystring");

 

First we create the HTTP server object, where request is an instance of http.ServerRequest, which contains details such as the requested URL, and response is an instance of http.ServerResponse, which is used to send a response to the client:

http.createServer(function (request, response) {
  ...
}).listen(8000);

 

The server can provide 2 different pages, the index page and a 404 page for all other requests.
As mentioned above the "request" object holds the url, but since we only want to know the requested page, we can use the url.parse().pathname function to get the requested page from the "request" object.
Requests for the index ('/') page will call the show_index() function, whereas any other request calls the show_404() function.

switch (url.parse(request.url).pathname) {
  case '/':
    show_index(request, response);
    break;
  default:
    show_404(request, response);
    break;
}

 

The index page displays a simple html document, including the requested query string.
First you need to server the header to the client:

function show_index(request, response) {
  sys.puts("Serving index page");
  response.writeHead(200, {'Content-Type': 'text/html'});
  ...
}

 

Then we prepare the output and extract the query from the request URL:
e.g.: http://127.0.0.1:8000/?test_arg=some_value would display "Request query: test_arg=some_value".

var output = 'node.js HTTP server exampleIndex output';
var url_request = url.parse(request.url).query;
output += "

Request query: " + url_request + "

";

 

Here comes the querystring sample, which deserialzies the url_request object making it easier to process the individual query arguments.
e.g.: http://127.0.0.1:8000/?test_arg=some_value would result in "Request query:

{"test_arg":"some_value"}"
if (url_request) sys.puts("Request query: " + JSON.stringify(querystring.parse(url_request)));
output += '';

 

Once the output has been prepared we can send it to the client and signal to the server that all of the response headers and body have been sent. The method response.end() *must* be called on each response!:

response.write(output);
response.end();

 

The 404 page simply sends to 404 header to the client and displays a short message:

function show_404(request, response) {
  sys.puts("Serving 404 error page");
  response.writeHead(404, {'Content-Type': 'text/plain'});
  response.write('404 - Please try again.');
  response.end();
}

 

Now lets take a look at a sample HTTP client, which connects to this server with a query argument and displays the returned content:
See 08_http/simple_client.js:

var sys = require("sys"),
   http = require("http");

var client = http.createClient(8000, "127.0.0.1");
var request = client.request("GET", "/?query_arg=testing", {"host": "127.0.0.1"});

sys.puts("Connecting to http://127.0.0.1:8000/?query_arg=testing");

request.addListener("response", function(response) {
  sys.puts("STATUS: " + response.statusCode);
  sys.puts("HEADERS: " + JSON.stringify(response.headers));
  response.setBodyEncoding("UTF8");
  response.addListener("data", function(chunk) {
    sys.puts("BODY: " + chunk);
  });
  response.addListener("end", function() {
    sys.puts("End of response");
  });
});
request.end();

 

The output will be something like:

Connecting to http://127.0.0.1:8000/?query_arg=testing
STATUS: 200
HEADERS: {"content-type":"text/html","connection":"close","transfer-encoding":"chunked"}
BODY: node.js HTTP server exampleIndex output

Request query: query_arg=testing

End of response

 

First we create the HTTP client and request objects, where the client contains the port and host you want to connect to and the request contains the method, path and host to connect to:

var client = http.createClient(8000, "127.0.0.1");
var request = client.request("GET", "/?query_arg=testing", {"host": "127.0.0.1"});

 

The request object's "response" event will emit when a response is received from the server. This event is emitted only once.
In this case we use it to display the returned status code, headers and set the encoding type of the response:

request.addListener("response", function(response) {
  sys.puts("STATUS: " + response.statusCode);
  sys.puts("HEADERS: " + JSON.stringify(response.headers));
  response.setBodyEncoding("UTF8");
  ...
}

 

The "data" event emits whenever data is received from the server:

response.addListener("data", function(chunk) {
  sys.puts("BODY: " + chunk);
});

 

The "end" event also emits exactly once per message and after it emitted no other events will be emitted on the response.

response.addListener("end", function() {
  sys.puts("End of response");
});

 

To finish sending the request use:

request.end();

 

These examples only scratched the surface of the available features in the HTTP module. For more advanced features and whole frameworks built around the http module, please see the node modules page: http://wiki.github.com/ry/node/modules

Node by example

Node by example: 7. net module

7. net module

The complete source code can be downloaded here: http://github.com/Hendrik/node-by-example

With the release of node v0.1.91, the previously existing "tcp" module has been renamed to "net". It provides 2 main objects, net.Server, which is used to create a TCP or UNIX server, and net.Stream, which is an abstraction of a TCP or UNIX socket.

Use

require('dns')

to access the features of this module.

 

The following is a tcp server and client example that listens on port 8000.
See 07_net/net_server.js

// based on http://nodejs.org/api.html#net-server-183

var sys = require("sys"),
    net = require("net");

var server = net.createServer(function(stream) {
  stream.setEncoding("utf8");
  stream.addListener("connect", function() {
    sys.puts("Client connected");
    stream.write("hello\r\n");
  });
  stream.addListener("data", function(data) {
    sys.puts("Received from client: " + data);
    stream.write(data);
  });
  stream.addListener("end", function() {
    sys.puts("Client disconnected");
    stream.write("goodbye\r\n");
    stream.end();
  });
});
server.listen(8000, "localhost");

// stop the server after 10 secs
setTimeout(function() {
  server.close();
}, 10000);

 

Create the server and make it listen to port 8000 on "localhost":

var server = net.createServer(function(stream) { ... });
server.listen(8000, "localhost");

 

Set the encoding of the stream:

stream.setEncoding("utf8");

 

The net.Stream object instances are an EventEmitter with the following events: 'connect': emmited when a stream connection is successfully established
'data': emmited when data is received, comes with a 'data' argument, which will be a Buffer or a String
'end': emmited when the other end of the stream sends a FIN packet

The following code will output "Client connected" to the console and send "hello\r\n" to the client upon a successful connection:

stream.addListener("connect", function() {
  sys.puts("Client connected");
  stream.write("hello\r\n");
});

 

Whenever the client sends data, the script will output "Received from client: [data]" to the console and send the sent data back to the client:

stream.addListener("data", function(data) {
  sys.puts("Received from client: " + data);
  stream.write(data);
});

 

When the client ends the connection "Client disconnected" will be sent to the console and a "goodbye\r\n" will be sent to the client.
stream.end() then sends a FIN packet to the client:

stream.addListener("end", function() {
  sys.puts("Client disconnected");
  stream.write("goodbye\r\n");
  stream.end();
});

 

Additionally the server will be closed automatically after 10 seconds:

// stop the server after 10 secs
setTimeout(function() {
  server.close();
}, 10000);

 

In order to run the full example please first start the server via

node net_server.js

and then start the client via

node net_client.js

Then monitor the output in both consoles, which will be similar to:

net_server.js: Client connected
net_client.js: Client connected.
net_client.js: Response from server: hello
net_client.js: Sent to server: close
net_server.js: Received from client: close
net_client.js: Response from server: close
net_client.js: Response from server: goodbye
net_server.js: Client disconnected
net_client.js: Disconnected from server

 

Let's see what the client does:
See 07_net/net_client.js

var sys = require("sys"),
    net = require("net");

var client = net.createConnection(8000);
client.setEncoding("UTF8");

client.addListener("connect", function() {
  sys.puts("Client connected.");
  // close connection after 2sec
  setTimeout(function() {
    sys.puts("Sent to server: close");
    client.write("close", "UTF8");
  }, 2000);
});

client.addListener("data", function(data) {
  sys.puts("Response from server: " + data);
  if (data == "close") client.end();
});

client.addListener("close", function(data) {
  sys.puts("Disconnected from server");
});

 

Create the client and make it connect to port 8000. Set the encoding type to utf8.

var client = net.createConnection(8000);
client.setEncoding("UTF8");

 

net.createConnection() uses "localhost" by default, but you can change it to a different host, e.g.: 127.0.0.1, using:

net.createConnection(8000, "127.0.0.1");

 

As mentioned in the server example, the stream object comes with the "connect", "data" and "close" event emitters.

In this case the client will send "Client connected." to the console after a successful connection has been established and then start a timer to close the connection automatically after 2 seconds:

client.addListener("connect", function() {
  sys.puts("Client connected.");
  // close connection after 2sec
  setTimeout(function() {
    sys.puts("Sent to server: close");
    client.write("close", "UTF8");
  }, 2000);
});

 

When the client receives data from the server it will send "Response from server: [data]" to the console and in case the sent data is "close", it will also (half) close the stream by sending a FIN packet to the server:

client.addListener("data", function(data) {
  sys.puts("Response from server: " + data);
  if (data == "close") client.end();
});

 

Once the connection has been closed, the client will output "Disconnected from server" to the console:

client.addListener("close", function(data) {
  sys.puts("Disconnected from server");
});

 

This is just a basic example of a tcp server & client app, please refer to the official documentation for further details: http://nodejs.org/api.html#net-server-183

Node by example

Node by example: 6. DNS Resolving

6. DNS Resolving

The complete source code can be downloaded here: http://github.com/Hendrik/node-by-example

The DNS module provides DNS resolving related features, such as IPv4 & IPv6 resolving, specific record types resolving and reverse resolving. [1]
Please note: even though the documentation mentions the lookup capability of MX record types, this is actually not supported at this time. (node v0.1.91)

Use

require('dns')

to access the features of this module.

 

The following is an example which resolves 'google.com' and then reverse resolves the returned IP addresses.

// taken from http://nodejs.org/api.html#dns-204

var sys = require("sys"),
    dns = require("dns");

dns.resolve4("google.com", function(err, addresses) {
  if (err) throw err;

  sys.puts("addresses: " + JSON.stringify(addresses));

  for (var i = 0; i < addresses.length; i++) {
    var a = addresses[i];
    dns.reverse(a, function (err, domains) {
      if (err) {
        sys.puts("reverse for " + a + " failed: " + err.message);
      } else {
        sys.puts("reverse for " + a + ": " + JSON.stringify(domains));
      }
    });
  }
});

output:

addresses: ["72.14.213.103","72.14.213.104","72.14.213.106","72.14.213.147","72.14.213.105","72.14.213.99"]
reverse for 72.14.213.99: ["pv-in-f103.1e100.net"]
reverse for 72.14.213.99: ["pv-in-f104.1e100.net"]
reverse for 72.14.213.99: ["pv-in-f106.1e100.net"]
reverse for 72.14.213.99: ["pv-in-f147.1e100.net"]
reverse for 72.14.213.99: ["pv-in-f105.1e100.net"]
reverse for 72.14.213.99: ["pv-in-f99.1e100.net"]

 

dns.resolve4(domain, callback) resolves a domain based on IPv4 queries.
You can use IPv6 queries with dns.resolve6(domain, callback).

dns.reverse(ip, callback) reverse resolves an IP address to an array of domain names.

Each DNS query can return an error code: dns.TEMPFAIL: timeout, SERVFAIL or similar. dns.PROTOCOL: got garbled reply. dns.NXDOMAIN: domain does not exists. dns.NODATA: domain exists but no data of reqd type. dns.NOMEM: out of memory while processing. dns.BADQUERY: the query is malformed.

The following is an example that resolves specific record types for the provided domain and checks for the dns.NODATA error code:

var sys = require("sys"),
    dns = require("dns"),
 domain = process.ARGV[2];

if (!domain)
  return sys.puts("Usage: node " + __filename.replace(__dirname + "/", "") + " domainname");

// resolve A entries
dns.resolve(domain, 'A', function(err, addresses) {
  if (err) {
    if (err.errno == dns.NODATA) sys.puts("No A entry for " + domain);
    else throw err;
  } else {
    sys.puts("addresses: " + JSON.stringify(addresses));
  }
});

// resolve TXT entries
dns.resolve(domain, 'TXT', function(err, addresses) {
  if (err) {
    if (err.errno == dns.NODATA) sys.puts("No TXT entry for " + domain);
    else throw err;
  } else {
    sys.puts("addresses: " + JSON.stringify(addresses));
  }
});

Run via:

node dns_2.js google.com

output:

addresses: ["72.14.213.99","72.14.213.103","72.14.213.105","72.14.213.106","72.14.213.147","72.14.213.104"]
addresses: ["v=spf1 include:_netblocks.google.com ip4:216.73.93.70/31 ip4:216.73.93.72/31 ~all"]

 

[1] http://nodejs.org/api.html#dns-204

Node by example

Node by example: 5. Child Process & File System

5. Child Process & File System

The complete source code can be downloaded here: http://github.com/Hendrik/node-by-example

Child Process: [1]
Please note:
at the time of this writing the api has changed and ChildProcess is now its own class, so please make sure you have the latest node version installed ( >=0.1.33 required ).
What was previously createChildProcess() is now require("child_process").spawn().

Node provides a tri-directional popen(3) facility through the ChildProcess class. It is possible to stream data through the child’s stdin, stdout, and stderr in a fully non-blocking way.

To create a child process use require("child_process").spawn().

See 05_ChildProc_FS/childprocess.js:

var    sys = require("sys"),
  filename = process.ARGV[2];

if (!filename)
  return sys.puts("Usage: node " + __filename.replace(__dirname + "/", "") + " filename");

// create child process and add a listener for its "output" event
var tail = process.createChildProcess("tail", ["-f", filename]);
tail.addListener("output", function(data) {
  sys.puts(data);
});
tail.addListener("exit", function(code) {
  sys.puts("Child process stopped with exit code: " + code);
});

// add a timer to kill the child proces after 10 seconds
setTimeout(function() {
  tail.kill();
}, 10000);

 

This is a small node app, which will "tail -f" a provided file through a child process and send the output to the command prompt. A timeout event will kill the child process after 10 seconds.

First we create the child process:

var tail = process.createChildProcess("tail", ["-f", filename]);

Then we attach event listeners for the "output" and "exit" events.
The "output" event is emitted every time the child process sends data to its stdout.
In this case we simply output the data to the comand prompt:

tail.addListener("output", function(data) {
  sys.puts(data);
});

The output can be handled as needed, e.g.: you could modify this app to send the output through a websocket to a browser.

After 10 seconds, the setTimeout function will kill the child process:

setTimeout(function() {
  tail.kill();
}, 10000);

This will then emitt the "exit" event with the final exit code of the child process:

tail.addListener("exit", function(code) {
  sys.puts("Child process stopped with exit code: " + code);
});

 

The child_process module also provides a high-level way to execute a command as a child process and buffer the output and return it in a callback via:

require("child_process").exec(command, callback)

The callback gets the arguments (err, stdout, stderr). On success "err" will be "null". On error "err" will be an instance of "Error" and "err.code" will be the exit code of the child process.

 

See 05_ChildProc_FS/childprocess_2.js:

var sys = require("sys"),
   exec = require("child_process").exec;

exec("ls /", function (err, stdout, stderr) {
  if (err) throw err;
  sys.puts(stdout);
});

 

File System: [2]

File I/O is provided by simple wrappers around standard POSIX functions. To use this module do require('fs'). All the methods have asynchronous and synchronous forms. Keep in mind that with the asynchronous methods there is no guaranteed ordering!

The following example code will: 1. create a new file "test_file.txt" 2. write data to the file 3. add a file watcher to it to monitor any changes 3. change the chmod value of the file 4. file watcher triggers due to the change 5. output the content of the current directory 6. output the content of "test_file.txt" 7. stop the file watcher 8. delete the file

1. create a new file: fs.open() opens a file, in this case for writing.

var file = path.join(__dirname, "test_file.txt");

fs.open(file, "w", 0644, function(err, fd) {
  if (err) throw err;
});

 

2. write data to the file: fs.write() allow you to write data to the file. Once the data has been written the file gets closed using the synchronous fs.closeSync() method.

  fs.write(fd, "Hello World", 0, "utf8", function(err, written) {
    if (err) throw err;
    fs.closeSync(fd);
  });

 

3. add a file watcher to it to monitor any changes: fs.watchFile() will monitor the provided file and triggers its callback whenever the file has been changed.

    fs.watchFile(file, function(curr, prev) {
      sys.puts("\n\ttest_file.txt has been edited");
      sys.puts("\tThe current mtime is: " + curr.mtime);
      sys.puts("\tThe previous mtime was: " + prev.mtime + "\n");
    });

 

4. change the chmod value of the file: Use fs.chmod() to change the chmod value of the specified file.

  fs.chmod(file, 0777, function(err) {
    if (err) throw err;
  });

 

5. output the content of the current directory fs.readdir() reads the entire content of the provided directory in an array excluding '.' and '..'.

  fs.readdir(__dirname, function(err, files) {
    if (err) throw err;
    sys.puts(JSON.stringify(files));
    show_file_content();
  });

 

6. output the content of "test_file.txt" fs.readFile() reads the content of the provided file.

  fs.readFile(file, function(err, content) {
    if (err) throw err;
    sys.puts(content);
  });

 

7. stop the file watcher To stop a file watcher simply use unwatchFile().

  fs.unwatchFile(file);

 

8. delete the file To delete a file use fs.unlink().

  fs.unlink(file, function(err) {
    if (err) throw err;
  });

 

See 05_ChildProc_FS/filesystem.js:

var sys = require("sys"),
   path = require("path"),
     fs = require("fs");

var file = path.join(__dirname, "test_file.txt");

fs.open(file, "w", 0644, function(err, fd) {
  if (err) throw err;
  sys.puts("File test_file.txt opened");
  fs.write(fd, "Hello World", 0, "utf8", function(err, written) {
    sys.puts("Data written");
    if (err) throw err;
    fs.closeSync(fd);

    fs.watchFile(file, function(curr, prev) {
      sys.puts("\n\ttest_file.txt has been edited");
      sys.puts("\tThe current mtime is: " + curr.mtime);
      sys.puts("\tThe previous mtime was: " + prev.mtime + "\n");
    });

    chmod_file();
  });
});

function chmod_file() {
  fs.chmod(file, 0777, function(err) {
    if (err) throw err;
    sys.puts("\nchmod value of test_file.txt set to 777");

    fs.chmodSync(file, 0644);
    sys.puts("chmod value of test_file.txt set to 644");

    show_dir();
  });
}

function show_dir() {
  sys.puts("\nContent of " + __dirname + ":");
  fs.readdir(__dirname, function(err, files) {
    if (err) throw err;
    sys.puts(JSON.stringify(files));
    show_file_content();
  });
}

function show_file_content() {
  fs.readFile(file, function(err, content) {
    if (err) throw err;
    sys.puts("\nContent of test_file.txt:");
    sys.puts(content);
    delete_file();
  });
}

function delete_file() {
  fs.unwatchFile(file);
  sys.puts("\nStopping watchFile of test_file.txt");

  fs.unlink(file, function(err) {
    if (err) throw err;
    sys.puts("\ntest_file.txt has been deleted.");
  });
}

 

[1] http://nodejs.org/api.html#child-processes-79
[2] http://nodejs.org/api.html#file-system-87

Previous Posts Next Posts