Recently I’ve been working with both the HTML 5 File API and Socket.IO and it occured to me that those technologies could be used to send, and push, files between the client and server. Immediately I set about making delivery.js, a simple node module that makes it extremely easy to send and push files between the client and the server via Socket.IO.

I should note that this is my first node module, so constructive critism and feedback are welcome and encouraged!

Example: Pushing An Image To The Client

Delivery.js on Github

In the following example, once a client connects to the server an image will be pushed to the client. When the client receives the image, the image’s data URL (“data:image/gif;base64,R0lGODlhEAAOALMAAOazT…”) is assigned to the src attribute of an img element.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$(function(){
  var socket = io.connect('http://0.0.0.0:5001');

  socket.on('connect', function(){
    var delivery = new Delivery(socket);

    delivery.on('receive.start',function(fileUID){
      console.log('receiving a file!');
    });

    delivery.on('receive.success',function(file){
      if (file.isImage()) {
        $('img').attr('src', file.dataURL());
      };
    });
  });
});

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

var io  = require('socket.io').listen(5001),
    dl  = require('delivery');

io.sockets.on('connection', function(socket){
  var delivery = dl.listen(socket);
  delivery.on('delivery.connect',function(delivery){

    delivery.send({
      name: 'sample-image.jpg',
      path : './sample-image.jpg'
    });

    delivery.on('send.success',function(file){
      console.log('File successfully sent to client!');
    });

  });
});

Advantages

While delivery.js is more of an experiment, there could be some advantages to using Web Sockets to transfer files. Once a Web Socket connection is established messages (frames) sent between the client and server contain only 2 additional bytes of overhead. In contrast, a traditional POST request, and response, may have headers totaling 871 bytes. This could be a significant addition if many files are being sent, and would be even more significant if files are being divided into batches before being sent to the server. When pushing files to the client, the overhead of traditional polling methods provides an even starker contrast to Web Sockets.

Disadvantages

The most apparent disadvantage would be the fact that it bipasses traditional caching methods. Instead of caching based on a file’s URL, caching would be based on the content of the Web Socket’s message. One possibility would be to cache a base64, or text, version of the file within Redis for fast, in memory, access.

Future

With zip.js files could be deflated, or inflated, within the browser. In-browser file zipping could have a positive impact on transfer speeds. To check the integrity of the unzipped file an md5 checksum could be generated client-side, using Paul Johnston’s JavaScript MD5 implementation, and then passed along with the zipped file.

For more documentation and examples see Delivery.js on Github:
Delivery.js on Github

I’m currently in the midst of creating an application that utilizes Socket.IO for real-time communication between users. Using mocha and should.js I was able to test objects within my node app. However, I quickly found that there were some odd synchronization issues between clients that the test cases couldn’t cover. To test the interaction between clients I needed a way to programatically communicate to my node server: Enter socket.io-client.

Testing Socket.IO on Github

Writing Tests For your Socket.io Application

As an example of how to test your socket.io application we’ll create a small chat application and test it using: socket.io-client, mocha and should.js.

Our chat server should be able to:

1. broadcast that a new user has joined
2. broadcast messages to the whole group
3. handle private messages between clients

If you hit any roadblocks see Trouble shooting at the bottom of this post.

Without further ado, below is the bare essentials for our socket.IO chat server (chat-server.js) and our tests (tests/test-chat-server.js).

1
2
3
4
5

var io = require('socket.io').listen(5000);

io.sockets.on('connection', function (socket) {

});

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

var should = require('should');
var io = require('socket.io-client');

var socketURL = 'http://0.0.0.0:5000';

var options ={
  transports: ['websocket'],
  'force new connection': true
};

var chatUser1 = {'name':'Tom'};
var chatUser2 = {'name':'Sally'};
var chatUser3 = {'name':'Dana'};

describe("Chat Server",function(){

});

First Test

Within the describe function (line 16 in tests) we will add our first test that determines if our chat server notifies all users that a new user has joined. It’s important to disconnect both clients once this test is over, otherwise the second test will trigger this test’s ‘new user’ event.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

it('Should broadcast new user to all users', function(done){
  var client1 = io.connect(socketURL, options);

  client1.on('connect', function(data){
    client1.emit('connection name', chatUser1);

    /* Since first client is connected, we connect
    the second client. */
    var client2 = io.connect(socketURL, options);

    client2.on('connect', function(data){
      client2.emit('connection name', chatUser2);
    });

    client2.on('new user', function(usersName){
      usersName.should.equal(chatUser2.name + " has joined.");
      client2.disconnect();
    });

  });

  var numUsers = 0;
  client1.on('new user', function(usersName){
    numUsers += 1;

    if(numUsers === 2){
      usersName.should.equal(chatUser2.name + " has joined.");
      client1.disconnect();
      done();
    }
  });
});

Now that our first test is complete, let’s run the server.

1

node chat-server.js

In another terminal window we can run the tests.

1

mocha -R spec

And both tests will fail (times out to be specific)! To pass the tests we simply update the chat-server.js to the following:

1
2
3
4
5
6
7
8

var io = require('socket.io').listen(5000);

io.sockets.on('connection', function (socket) {
  socket.on('connection name',function(user){
    io.sockets.emit('new user', user.name + " has joined.");
  })

});

When you run the tests again they should both pass.

Second Test

Now, we’ll add the second test that determines if a message sent from one client is sent to all the clients connected to the chat server. It may not initially be obvious why I’m using 3 clients, but it should be in the 3rd test. To determine if all 3 clients have received a message we keep a counter (messges) and check that each received message is the correct message.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

it('Should be able to broadcast messages', function(done){
  var client1, client2, client3;
  var message = 'Hello World';
  var messages = 0;

  var checkMessage = function(client){
    client.on('message', function(msg){
      message.should.equal(msg);
      client.disconnect();
      messages++;
      if(messages === 3){
        done();
      };
    });
  };

  client1 = io.connect(socketURL, options);
  checkMessage(client1);

  client1.on('connect', function(data){
    client2 = io.connect(socketURL, options);
    checkMessage(client2);

    client2.on('connect', function(data){
      client3 = io.connect(socketURL, options);
      checkMessage(client3);

      client3.on('connect', function(data){
        client2.send(message);
      });
    });
  });
});

If you run the tests again it should fail! We can fix this by updating the server code to look like:

1
2
3
4
5
6
7
8
9
10
11

var io = require('socket.io').listen(5000);

io.sockets.on('connection', function (socket) {
  socket.on('connection name',function(user){
    io.sockets.emit('new user', user.name + " has joined.");
  });

  socket.on('message', function(msg){
    io.sockets.emit('message', msg);
  });
});

Third Test

With the first two bits of functionality complete, and tested, we now need to test that 1) a client can send a private message to another client and 2) no other clients receive that message. At minimum we need 3 clients to test this condition. Once all 3 clients have connected, the 3rd client will send a private message to the first client. It’s easy to test that the first client has received the message, but how do we test that the 2nd client has not? The solution I’m using below is that the test simply waits 40 milliseconds and if only 1 message has been received, there’s a strong chance that client 2 has not received a message. I’m not sure how to write the test to be 100% sure that the client has not received the message - any thoughts?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

it('Should be able to send private messages', function(done){
  var client1, client2, client3;
  var message = {to: chatUser1.name, txt:'Private Hello World'};
  var messages = 0;

  var completeTest = function(){
    messages.should.equal(1);
    client1.disconnect();
    client2.disconnect();
    client3.disconnect();
    done();
  };

  var checkPrivateMessage = function(client){
    client.on('private message', function(msg){
      message.txt.should.equal(msg.txt);
      msg.from.should.equal(chatUser3.name);
      messages++;
      if(client === client1){
        /* The first client has received the message
        we give some time to ensure that the others
        will not receive the same message. */
        setTimeout(completeTest, 40);
      };
    });
  };

  client1 = io.connect(socketURL, options);
  checkPrivateMessage(client1);

  client1.on('connect', function(data){
    client1.emit('connection name', chatUser1);
    client2 = io.connect(socketURL, options);
    checkPrivateMessage(client2);

    client2.on('connect', function(data){
      client2.emit('connection name', chatUser2);
      client3 = io.connect(socketURL, options);
      checkPrivateMessage(client3);

      client3.on('connect', function(data){
        client3.emit('connection name', chatUser3);
        client3.emit('private message', message)
      });
    });
  });
});

Once again, running the tests should fail. Lets add code to enable private messages. Ideally we’d use Redis Pub/Sub to simplify message passing, but for the sake of brevity we’ll omit it here. We’ll keep track of clients and their sockets using the clients hash. The hashe’s key will be the client’s name, and the value will be the client’s socket. When the client disconnects we delete that entry from the hash.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

var io = require('socket.io').listen(5000);
var clients = {};

io.sockets.on('connection', function (socket) {
  var userName;
  socket.on('connection name',function(user){
    userName = user.name;
    clients[user.name] = socket;
    io.sockets.emit('new user', user.name + " has joined.");
  });

  socket.on('message', function(msg){
    io.sockets.emit('message', msg);
  });

  socket.on('private message', function(msg){
    fromMsg = {from:userName, txt:msg.txt}
    clients[msg.to].emit('private message', fromMsg);
  });

  socket.on('disconnect', function(){
    delete clients[userName];
  });
});

Congratulations, you have a node.js chat server and you’ve tested it!

Testing Socket.IO on Github

Trouble Shooting

• Have you installed node.io-client, mocha and should?
• Did you install any of the modules locally, but in another folder? If so try doing a global install.
• Make sure you restart the server before each test run.
• Running mocha -R spec will not work within the test folder, this command looks for the test folder, so cd .. if you’re in the test folder.
• The version of socket.io-client that was in the npm repositories at the time of writing this would not connect to the node server. To get the latest version I ran: npm install https://github.com/LearnBoost/socket.io-client/zipball/master

Updates

Using coffeescript @jamescarr refactored the above code so the server starts automatically during testing: socket.io-test

With the public release of ember.js, I excitedly poured over the examples hoping to find a framework that would make Creating, Reading, Updating and Deleting (CRUD) easier. When push comes to shove, the vast majority of apps that I’ve worked on involve CRUD.

I want to see the following (excuse the pseudo code):

1
2
3

model({
  validation: function(m){...}
});

1
2
3
4
5
6
7
8
9
10
11
12

controller({
  show: function(){...},

  index: function(){...},

  new: function(){...},

  create: function(){...},

  ...

});

1
2
3
4

<form action=''>
  <input type='text' name='firstName'>
  <input type='submit' value='submit'>
</form>

Will your framework help me 1) write less code, 2) get things done faster and 3) with less errors? Show me.

PS: I don’t want to diminish the tremendous effort it takes to create a framework and release it, but please go the extra mile and provide some concrete CRUD examples for a single class!

I want to master JavaScript. Before humbly doing so, I think it’s important to define what I think mastering JS consists of. In my opinion there are two parts to mastering a programming language. The first is mastering the language itself: syntax, important functions, classes, idioms, etc. The second part is mastering the programming paradigm, or paradigms, that the language falls in. This second part is difficult with JS, as it is object oriented, functional and imperative . Thus, to master JS I must master both parts: 1) the language and 2) the paradigms.

Before explaining how I’m going to accomplish this goal, it’s worth while for me to briefly explain why I want to. I have been programming for 4 years and I just recently finished a computer science degree. In school I completed assignments where I learned elements from the 3 major programming paradigms that JS falls into. Likewise, I spent the last year using JS, but I have not delved deeper into it. Since I have scratched the surface I have realized that I enjoy creating web applications, but I eagerly want to be better at it.

Becoming a master will no doubt take years of effort, and countless hours, however I think the first four months will be critical - they will provide the foundation to build my knowledge upon. In the next four months I hope to accomplish the following (roughly in order):

1. Read JavaScript: The Good Parts, by Douglas Crockford. (I’ve started this part)
2. Examine jQuery’s source code.
3. Contribute to jQuery, most likely a bug fix.
4. Read JavaScript Web Applications, by Alex MacCaw.
5. Examine Backbone.js’s source code .
6. Contribute to Backbone.js

While completing those six goals I plan on weaving what I’ve learning into projects that I’ll be undertaking in the next four months. My hope is that by reading books, and other people’s code, I’ll learn how to write better code and engineer larger JS applications. Furthermore, by actually working on projects I’ll reinforce what I’ve learned and I’ll have something tangible to show.

Is this the best way of accomplishing my goal of mastering JS? Discuss on Hacker News.

After 3 years without change, I’ve redone my personal website using Jekyll and Bootstrap. The previous version of liamkaufman.com was hosted on Google App Engine and consisted of a very basic CMS that I made before I knew what I was doing. I was both unhappy with the look of the site, and the code behind it. At the same time I was interested in using Jekyll, as I wanted to avoid the hassles associated with database backed sites. I also appreciate the fact that my site, and it’s content are easy to deploy and update via Github.

Overall the process of learning Jekyll and Bootstrap was much easier than anticipated. To learn Jekyll I examined the source code behind Scott Chacon’s blog. Learning Bootstrap consisted of referencing CSS classes in my DOM and referencing the Bootstrap CSS file itself: not much to it.

In an effort to add some personality to this site I thought I’d list my favourite songs. iTunes provides a mechanism for users to rate songs, by giving a song 0 to 5 stars. After years of listening to music, and rating it, I have a large collection of rated songs. Manually copying and pasting each song into an HTML file seemed inefficient and time consuming, enter Python.

In an effort to learn Python, and parse my iTunes library, I created pyItunes, a Python module. pyItunes includes three classes that can parse the “iTunes Music Library.xml” file and convert it into list of Song objects. I decided to roll my own micro-xml parser that can only parse the iTunes xml file. Although, reinventing the wheel, especially an XML parser, is likely a horrible idea I felt it would be a good learning experience. In future versions I may swap an existing XML parser that is much faster and more efficient than the one I created.

Download pyItunes

pyItunes Usage