Code documentation - express-examples/sqlite

State of the art

Code documentation appears to be not so clear into JS world. I've been reading several pages and blogs about this, and finally I got 2 main solutions, jsdoc-toolkit and ScriptDoc.

At the beginning I leaned for ScriptDoc, because people say is very similar to Javadoc and some days ago I developed a mobile app using Titanium. But, I found JSDoc more mature, even when the project appears to be forgotten (last commit on June 2010).

Finally, I selected JSDoc. Here is a list of people using it. The project documentation here and a list of tags here.

An example

From /lib/rest/common.js

/** 
* @fileOverview Common JS for REST services (express-examples/sqlite)
*
* @author Rodolfo Campos <camposer at gmail dot com>
* @version 1.0
*/

/**
 * HTTP codes 
 * @see For all HTTP codes refer to: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html 
 */
var HTTP_CODE_OK = 200
  , HTTP_CODE_UNAUTHORIZED = 401
  , HTTP_CODE_FORBIDDEN = 403
  , HTTP_CODE_NOT_FOUND = 404
  , HTTP_CODE_METHOD_NOT_ALLOWED = 405
  , HTTP_CODE_PRECONDITION_FAILED = 412;

/** 
 * JSON Content-Type HTTP Header
 * @field
 */
var JSON_RESPONSE_HEADER = {'Content-Type': 'application/json; charset=utf-8'};

/** 
 * Writes HTTP code 200 and Content-Type for JSON in HTTP response headers
 * @function 
 * @param {HttpResponse} res HTTP response 
 * @param {Number} code HTTP code to be written in header
 */
var ok = function(res) {
  res.writeHead(HTTP_CODE_OK, this.JSON_RESPONSE_HEADER);
};

/** 
 * Writes HTTP code 4XX (error) and Content-Type for JSON in HTTP response headers
 * @function 
 * @param {HttpResponse} res HTTP response 
 * @param {Number} code HTTP code to be written into header
 */
var nok = function(res, code) {
  res.writeHead(code, JSON_RESPONSE_HEADER);
};

/**
 * Executes validate method using params, if true, calls execute method, if not, returns HTTP code 412
 * @param {HttpResponse} res HttpResponse object
 * @param {Mixed|Object} params Can be value or object (JSON)
 * @param {Function([params])} validate Function used for validation, if returns true everything is OK, 
 *   if not something went wrong and call returns HTTP_CODE_PRECONDITION_FAILED(412)
 * @param {Function(params, callback)|Function(callback)} execute Function executed if validate=true. 
 *   Parameters (params) can be value (e.g. String) or Object (JSON). Callback should receive 
 *   one parameter (result) 
 *   
 * @example 
 * common.call(
 *   res, 
 *   params.name, 
 *   function(name) { 
 *     var valid = true;
 *
 *     if (name == null) 
 *       valid = false; 
 * 
 *     return valid;
 *   },
 *   function(params, callback) {
 *     // ...
 *     // Get result
 *     callback(result);
 *   }
 * );
 */
this.call = function(res, params, validate, execute) {
  if (validate(params)) {
    if (params != null) {
      execute(params, function(result) { // TODO: Add extra parameter for error msg
        if (result) 
          ok(res); 
        else 
          nok(res, HTTP_CODE_FORBIDDEN); 

        res.end(JSON.stringify(result)); 
      });
    } else {
      execute(function(result) { // TODO: Add extra parameter for error msg
        if (result) 
          ok(res); 
        else 
          nok(res, HTTP_CODE_FORBIDDEN); 

        res.end(JSON.stringify(result)); 
      });
    }
  } else {
    nok(res, HTTP_CODE_PRECONDITION_FAILED); 
    res.end(); 
  }
};

Generating documentation

For generating the HTML documentation corresponding to the example above (common.js), you just have to download jsdoc-tool from here, unzip it, and run the following command:

$ java -jar jsrun.jar app/run.js -a -t=templates/jsdoc -d=common common.js

Where:

• jsrun.jar: Generator
• app/run.js: Bootstrap script
• -a: Include all functions, even undocumented ones
• templates/jsdoc: Directory of the templates to be used
• -d: Directory where output is going to be put. By default ./out folder
• common.js: File parsed

For JsDoc inline documentation use:

$ java -jar jsrun.jar app/run.js --help

You can use -r option for generating documentation recursively, specifying a directory to be explored. I like this option, but specifying directories, for example:

$ java -jar jsrun.jar app/run.js -a -t=templates/jsdoc -d=sqlite -r ~/express-examples/sqlite/lib/rest ~/express-examples/sqlite/lib/service

Note: If you specify the root directory as starting point, you're going to have a mess with all JS files in the project (including node_modules).

Updated code documentation can be found inside the project at /jsdoc folder.

REST - express-examples/sqlite

I've developed a very simple RESTful app, only considering CRUD operations over an user entity.

Naming convention

I followed some good recommendations mentioned in this article, shared by jjeronimo.

In general, the API is:

• getUsers. Retrieves all users (SELECT * FROM user).

curl -X GET http://localhost:3000/rest/users

• getUsersById. Retrieves an user by its id (SELECT * FROM user WHERE id = ?).

curl -X GET http://localhost:3000/rest/users/1

• addUser. Adds an user (INSERT INTO user(name) VALUES(?)).

curl -X PUT -H "Content-type:application/x-www-form-urlencoded" -d "name=rodolfo" http://localhost:3000/rest/users

• updateUser. Updates an user (UPDATE user SET name = ? WHERE id = ?).

curl -X POST -H "Content-type:application/x-www-form-urlencoded" -d "id=1&name=juan" http://localhost:3000/rest/users

• removeUser. Removes an user by its id (DELETE FROM user WHERE id = ?).

curl -X DELETE -H "Content-type:application/x-www-form-urlencoded" -d "id=1" http://localhost:3000/rest/users

As you can see, the URLs follow the pattern: rest/users. Some comments about this:

• URLs start with the word "rest" for easily identifying them as part of the REST API. Another part indicating the API version can be added, like: rest/v1/users (util for backwards compatibility issues).
• A REST API should be easy to discover, just adding and removing parameters. For example: if you want all users can use /users, but if you want an specific user can use /users/1. Maybe a better option, for more complex cases, could be: /users/id/1, because this way results "evident" something like: /users/name/rodolfo/status/0. However, I prefer to specify getXxxById following the entity name (written in plural) by its id.

Facade-Service

I think REST APIs should be coded using Facades. The idea is to decouple business logic into some classes, and treat REST, SOAP, and whatever interfaces aside.

Following an example extracted from /lib/rest/user.js:

var common = require('./common');

var userService = require('../service/user');

this.addUser = function(req, res) {
  var params = req.body;

  common.call(
    res, 
    params.name, 
    function(name) { 
      var valid = true;

      if (name == null) 
        valid = false; 
  
      return valid;
    },
    userService.newInstance().addUser
  );
};

// ...

As you can see in the code above, inside the REST addUser method there are only operations related with parameters filling and validations. For this specific implementation, I coded a generic method for calling services (common.call). But you could just take parameters, validate them, call the corresponding service method, and return the expected output; sort of controller in a MCV implementation.

Call method for integrating Service - REST Facade

I wrote a method call (lib/rest/common.js) for integrating Service methods and REST facades. Following the code main highlights:

// ...

this.call = function(res, params, validate, execute) {
  if (validate(params)) {
    if (params != null) {
      execute(params, function(result) { // TODO: Add extra parameter for error msg
        if (result) 
          ok(res); 
        else 
          nok(res, HTTP_CODE_FORBIDDEN); 

        res.end(JSON.stringify(result)); 
      });
    } else {
      execute(function(result) { // TODO: Add extra parameter for error msg
        if (result) 
          ok(res); 
        else 
          nok(res, HTTP_CODE_FORBIDDEN); 

        res.end(JSON.stringify(result)); 
      });
    }
  } else {
    nok(res, HTTP_CODE_PRECONDITION_FAILED); 
    res.end(); 
  }
};

// ...

The call method executes validate method using params, if true, calls execute method, if not, returns HTTP code 412 (HTTP_CODE_PRECONDITION_FAILED).

• res: HttpResponse object
• params: Can be value or object (JSON)
• validate: Function used for validation, if returns true everything is OK (HTTP_CODE_OK=200), if not, something went wrong and returns HTTP_CODE_PRECONDITION_FAILED(412)
• execute: Function executed if validate=true. Parameters (params) can be value (e.g. String) or Object (JSON). Callback should receive one parameter (result).

An example about how to use the common.call method can be found above (this.addUser).

Init script - express-example/sqlite

I've been reading in the Express Forum about deploying Node.js applications and found some interesting points of view about that. My proposal is to deploy using Capistrano. I hope to comment about it in this space soon.

The thing that takes me to this space now is related to init scripts... Some days ago I was "fighting" against Pentaho BI Server init scripts, and based on my previous work, I propose the following script (init.d/sqlite) for Node.js applications. For installing, you need to change the following variables:

SQLITEAPP_DIR="/home/rodolfo/workspace/express-examples/sqlite"
SQLITEAPP_LOG="$SQLITEAPP_DIR/log/sqlite.log"

After changing that variables to the root directory where you have the Express App, and where you want to leave logs, can just start or stop the server like this:

$ bash init.d/sqlite start
$ bash init.d/sqlite stop

If you want to include the script into system init.d scripts (for -maybe- adding them later to rcX.d scripts) can do the following:

$ chmod +x init.d/sqlite
$ sudo ln -s init.d/sqlite /etc/init.d/sqlite-express-example

Now for starting and stopping:

$ sudo /etc/init.d/sqlite-express-example start
$ sudo /etc/init.d/sqlite-express-example stop

NOTE: As you can see, my script has the name of the app, that's because I'm assuming one app per Node.js instance, but if you're going to have only one instance for all apps, can rename all variables and the script to something related to Node.js.

ERRATUM: Is obvious that the name sqlite for the script is a very bad choice. Why I put that name? Nevermind... You can easily change it.

DAO - express-examples/sqlite

According to Wikipedia, a DAO "is an object that provides an abstract interface to some type of database or persistence mechanism, providing some specific operations without exposing details of the database. It provides a mapping from application calls to the persistence layer".

This concept just introduces more abstraction and decoupling into your code. In my experience it results fundamental for maintaining painless an app.

Following, an implementation example of this pattern (lib/dao/user.js):

var sqlite3 = require('sqlite3').verbose();
var db = new sqlite3.Database('./db/db.sqlite');

this.newInstance = function() {
  return new impl();
};

function impl() {
  // ...

  this.getUsers = function(callback) {
    db.all("SELECT rowid AS id, name FROM user", function(err, rows) {
      if (err) {
        console.log(err);
        rows = null;
      }

      callback(rows);
    });
  };

  // ...
};

As you can see in the code above, we have a class with several methods (specially getUsers). If you want to query the database, you just have to call the necessary method. The rule is simple, group database operations in related classes (e.g. user) and never put a query outside that scope.

Another important thing is: DAO's shouldn't do anything else than database operations, no validation, no business logic! Validations (extra app forms) and "Business Logic" should be put in other classes... You can call them: logics, services, or whatever you want, but separated. Here is an example of a service class (lib/service/user.js):

var userDao = require('../dao/user');

this.newInstance = function() {
  return new impl();
};

function impl() {
  // ...

  this.getUsers = function(callback) {
    userDao.newInstance().getUsers(callback);
  };

  // ...
}

In this example, the service method getUsers doesn't include any business logic, but maybe this can change in the future and for that reason is better to keep it separated.

And finally, you can call the services from a controller or service facade (REST, SOAP, ...) like (lib/rest/user.js and lib/rest/common.js):

var userService = require('../service/user');

this.getUsers = function(req, res) {
  common.call(
    res, 
    null, // No params
    function(params) { return true },
    userService.newInstance().getUsers
  );
};

Classes and Factories - express-examples/sqlite

In JS there are several ways of defining and instantiating a class. A really good article commenting ways of doing that, courtesy of jjeronimo, available here.

Based on the article mentioned above, and my previous experience with factories implementations in Java (an example), I propose to implement classes this way (lib/dao/user.js):

var sqlite3 = require('sqlite3').verbose();
var db = new sqlite3.Database('./db/db.sqlite');

this.newInstance = function() {
  return new impl();
};

function impl() {
  // ...

  this.getUserById = function(id, callback) {
    db.all("SELECT rowid AS id, name FROM user WHERE id=?", [ id ], function(err, rows) {
      if (err) 
        console.log(err);

      callback((rows)?rows[0]:null);
    });
  };

  // ...
};

If you want to use the class defined above, you just have to do something like (lib/service/user.js):

var userDao = require('../dao/user');

  // ...

  this.getUserById = function(id, callback) {
    userDao.newInstance().getUserById(id, callback);
  };

  // ...

One of the main advantages of this approach, factories based, is that you can mockup very easy your classes for testing. For example:

this.newInstance = function(env) {
  if (env && env == 'test') 
    return new mock();
  else
    return new impl();
};

function impl() {
  // Normal implementation
}

function mock() {
  // Mockup implementation (for testing)
}

Then, if you want to test, you just have to specify the parameter test when instantiating the object, something like:

var service = require('service');
var serviceMock = service.newInstance('test');
// Test ...

var serviceReal = service.newInstance();
// Do something 'real'

SQLite - express-examples/sqlite

Installing SQLite3

SQLite3 is very simple, and perfect for examples "plug & play". For installing it:

$ sudo aptitude install sqlite3

Creating a Database

In my case I just entered:

$ sqlite3 db/db.sqlite

Once inside, you can enter SQL commands. Following sqlite example table:

sqlite> CREATE TABLE user (name VARCHAR(50)); 

In SQLite exists a default auto-increment field named `rowid`, that means you can make queries like:

sqlite> SELECT rowid AS id, name FROM user WHERE id=?

Installing SQLite plugin for Node.js

I'm using node-sqlite3. I found this plugin immature and not working properly under several scenarios, since I'm not really interested in developing something complex with sqlite, not going to invest more time on this, but maybe node-sqlite is a better option.

For installing it:

$ npm install sqlite3

No matter which of the 2 previously mentioned plugins you use, you have to use callback programming style.

var sqlite3 = require('sqlite3').verbose();
var db = new sqlite3.Database('./db/db.sqlite');

// Querying
db.all("SELECT rowid AS id, name FROM user WHERE id=?", [ id ], function(err, rows) {
  if (err) 
    console.log(err);

  callback((rows)?rows[0]:null);
});

// Adding
db.run("INSERT INTO user(name) VALUES(?)", [ name ]/*, function(error) {
  if (error) 
    console.log(error);

  callback((error)?false:true);
}*/);
callback(true); // TODO: Callback is not working!!

As you can see in the code above, callbacks are not working properly in `db.run` method. But, the really interesting thing I want to emphasize is the use of them (callbacks), this adds a-synchronicity to your code, which means things are not blocked until getting a response. Of course if you're experienced with thread capable languages (e.g. Java), this may not surprise you, but if you're PHP programmer you should be surprised. Another thing, the simplicity and let me say "harmony" of this approach is outstanding.

Routing - express-examples/sqlite

Changes for routing

When you install express it adds routes inside app.js, I'm not really comfortable with that because this file for me must be only for app configuration, and things like routes, which changes a lot, should be configured in other place. My changes were:

• Remove all routes mappings from app.js and add one call for global routes: routes.route(app);
• Leave calls for mappings inside routes/index.js separated in files. For example:

var views = require('./views');
var rest = require('./rest');

exports.route = function(app) {
   views.route(app);
   rest.route(app);
};

• Add route method for each group of routes (this can be splitted in more files for larger apps). For example (routes/rest.js):

var userRest = require('../lib/rest/user');
var BASE_URL = '/rest';

this.route = function(app) {
  this.users(app);
};

this.users = function(app) {
  app.get(BASE_URL + '/users', userRest.getUsers);
  app.get(BASE_URL + '/users/:id', userRest.getUserById);
  app.put(BASE_URL + '/users', userRest.addUser);
  app.delete(BASE_URL + '/users', userRest.removeUser); 
  app.post(BASE_URL + '/users', userRest.updateUser);
};

Note: I propose this files remain without any "business" logic, just used for routing purposes.

Some words about MVC

In this case I didn't use MVC, because my approach is for heavier clients, doing Ajax requests to RESTful applications. Something code-behind oriented. But... What if you don't want that? Well, here is a proposal (based on my actual code) for MVC:

• Change routes/views.js like this:

var userController = require('../lib/controller/user');

this.route = function(app) {
  app.get('/', userController.index);
};

Note: You can divide route function like in routes/rest.js

• Add /lib/controller/user.js like this:

var userService = require('../service/user');

this.index = function(req, res){
  var users = userService.getUsers();
  res.render('index', { title: 'List of users', session: req.session, users: users });
}

• Inside index.ejs render users:

// ...

<% for (var i=0; users.length; i++) { %>
    <tr>
      <td><%= users[i].id %></td>
      <td><%= users[i].name %></td>
    </tr>
<% } %>

// ...

HTML escaped using: http://www.htmlescape.net/htmlescape_tool.html

Setup - express-examples/sqlite

Following the steps I performed for creating the project, after being installed node.js and npm (see [How to run it](wiki/How to run it) for details on how to install them):

Installing express

I wanted to install express globally, for doing that:

$ sudo npm install -g express

Notes:

• If you want to install an specific version (in my case I have 3.0.0beta7) you can use name@version, e.g. $ sudo npm install -g express@3.0.0beta7
• If you don't want a global version of express, you can remove -g option. Packages "not global" are installed into node_modules folder.

Creating the application skeleton

The application skeleton was created with express, specifying 2 "not standard" options:

• EJS. As view engine. I prefer it a lot compared with Jade, which is the default for express, because I'm more familiar with HTML, and really don't think jade's way to implement web pages is realistic and maintainable.
• Session. Just enabled the use of sessions within express' application.

Following the command used:

$ express --sessions --ejs

How to run the project - express-examples/sqlite

Installing Node.js

For create the application I installed node.js and its package manager npm. For doing that I just typed:

$ sudo aptitude install nodejs npm

My actual linux installation (obsolete):

$ cat /etc/issue
 Ubuntu 10.10 \n \l

My actual node.js packages are:

$ dpkg -l node\* npm
...
ii  nodejs                      0.8.2-1chl1~maverick1       Node.js event-based server-side javascript engine
ii  nodejs-dev                  0.8.2-1chl1~maverick1       Development files for Node.js
ii  npm                         1.1.39-1chl1~maverick1      package manager for nodejs

Note: If you want to install an specific version, you can do it directly from the sources.

Downloading the code

$ git clone https://github.com/camposer/express-examples.git

Starting the server

$ cd express-examples
$ cd sqlite
$ node app
Express server listening on port 3000

Running the app

Just enter in your browser: http://localhost:3000

Testing the REST services

You can test the REST services directly using CURL (installing it is as simple as enter: `sudo aptitude install curl`). Inside the app there is a script named curltest.sh.

echo "getUsers"
curl -X GET http://localhost:3000/rest/users; echo

echo "getUserById"
curl -X GET http://localhost:3000/rest/users/1; echo

echo "addUser"
curl -X PUT -H "Content-type:application/x-www-form-urlencoded" -d "name=rodolfo" http://localhost:3000/rest/users; echo

echo "updateUser"
curl -X POST -H "Content-type:application/x-www-form-urlencoded" -d "id=1&name=juan" http://localhost:3000/rest/users; echo

echo "deleteUser"
curl -X DELETE -H "Content-type:application/x-www-form-urlencoded" -d "id=1" http://localhost:3000/rest/users; echo

Kickoff - express-examples/sqlite

This is a little late kickoff for the project, but as I'm starting this blog and planning to bring here all my previous articles related...

Project location:

You can find the project in my github space at: github.com/camposer/express-examples

I'm planning to bring here all articles previously wrote in the project's wiki, but while I do that, you can find project's documentation here

Project goal:

I've been playing with Node.js for several days and started to ask myself how an application for real environments should be... This is how I got here, wanting to clarify common things/problems such as: i18n, MVC, DAO, SOA, REST, testing (xUnit style), documentation (Javadoc style), etc.

I hope this "conventions" (set of technologies and practices) help other programmers facing similar problems. I'm planning to add other examples including NoSQL and ORM, but right now only developed a simple app with Express as web framework and SQLite as SQL database engine. The app has CRUD operations for a user entity and was developed the way I think web apps should be programmed this days:

• Presentation: HTML and CSS (inside CSS files). The HTML files should contain only static data and controls, NO format and NO logic.
• Presentation logic: Javascript (inside JS files) and jQuery. jQuery Template for modifying presentation dynamically using HTML blocks. jQuery UI for improve user interface and effects.
• Business logic interface: REST services. Presentation logic call REST services (using jQuery), which produces JSON data.
• Business logic: Logic or service classes, separated from REST facades. This way your app can scale and be maintained easily. SOA style, decoupling!!!
• Data logic: DAO classes. If you include an ORM, DAO should exist also for abstraction.

Editorial

I usually write my technical findings and thoughts in 2 blogs, camposer-techie.blogspot.com and 21red.es. This blogs are written in spanish and its main purpose is to share knowledge, contributing for mitigate the digital divide through the sharing of quality contents in spanish.

Recently I've created a project in my space at github and started documenting it using github's wiki. So far, the tool was ok for me, but right know I'm missing some functionalities and wanted to write different things in english, which is the lingua-franca in the IT world.

As you may know or noticed, I'm not a native-english speaker, so, for sure you're going to find some mistypings and grammatical errors in my posts... sorry in advance :-)