JavaScript Filter, Map, Reduce in 5 Minutes

This post will summarize JavaScript’s filter, map, and reduce methods with simple examples.

Filter

From a given array of numbers, we want to keep the even numbers only. In other words, we want to filter the array. Code:

console.log(
	[1,2,3,4,5,].filter(function(num){
		return num % 2 === 0;
	})
);

The argument to filter is a function that tests if the current element of the array passes a condition; in our case, if it is even. If it passes the condition, the element is added to another array. After all the elements have been checked, the newly generated array is returned. The original array stays intact.

Map

From a given set of temperature readings, we want to convert the temperature in Kelvin to either Celsius or Farenheit. In other words, we want to map the Kelvin value to the corresponding Celcius or Farenheit value. JavaScript’s map method does just that. It takes in a function that does the transformation and returns an array of transformed values. Code:

var readings = [562,881,996,662];
var toCelsius = function(temp){ return temp - 273; }
var toFarenheit = function(temp){ return (temp-273) * 1.8 + 32; }
console.log(readings.map(toCelsius));
console.log(readings.map(toFarenheit));

Reduce

Given an array of numbers, we want to find its sum. In other words, we want to reduce it to its sum. Code:

  
console.log(
	[1,2,3,4,5].reduce(function(total,num){ 
		return total += num;
	},0)
);

The reduce method takes two parameters – a function that performs the reduction and an initial value. The reduction function is passed two values by reduce – the intermediate sum (total) and the current element (num). We return the sum of intermediate sum and current number. At the end, our array gets reduced to its sum.

JavaScript Filtering Array Contents

In this post I will show you can filter the contents of an array – keep only the elements that meet a specific criteria. I will first show you how you can implement this yourself and then show you how you can use the in-built filter method of the array. Here we go:

function filter(array,test){
	var passed = [];
	for(var i = 0; i < array.length; i++){
		if(test(array[i]))
			passed.push(array[i]);
	}
	return passed;
}
var colors = ["red","green","blue","black"];
console.log(filter(colors,function(color){
	return color.charAt(0) === "b";
}));

The filter function takes an array and a function that tests an element of the array. If that element passes the test, it is added to passed. The filter function does not modify the original array. Filtering is such a common task that arrays have a predefined filter that takes in a function that checks the array element that was passed to it. Here is how it works:

var colors = ["red","green","blue","black"];
console.log(colors.filter(function(color)){
	return color.charAt(0) === "b";
});

That’s it on filtering.

JavaScript Higher Order Functions

In this post I will be talking about higher order functions in JavaScript and how they work. For a function to be a higher order function, it must meet atleast one of the criteria below:

  1. It takes a function as an argument
  2. It returns a function

In the last post about closures and lexical scoping, we created a function greaterThan that returned another function. The greaterThan function is a higher order function. This is what it looks like:

function greaterThan(m){
    return function(n){ return n > m; }
}

All the functions that return closures are, by definition, higher order functions. So what’s the benefit of using higher order functions? Well, many.

First, they help in abstracting the code. One has to traverse arrays a lot of times in programming. The usual way is to write a loop and maintain a counter. There is nothing wrong with this approach but one can certainly do better. JavaScript has an in-built forEach function that abstracts the array traversal for you.

var arr = [1,2,3,4,5,6];
arr.forEach(function(n){
	if(n%2===0) console.log(n);
});

The forEach function hides the array traversal from you and only asks you to specify what action needs to be taken on each item of the array. You pass in a function that specifies the action that needs to be taken. As forEach begins stepping through the array, it passes the element to your function which decides what needs to be done. This also reduces the number of lines you need to code as you do not have to write the same loop over and over again.

Second, they let you create your own control flows. Consider the example taken from the book “Eloquent JavaScript”:

function unless ( test , then ) {
	if (! test ) then () ;
}
function repeat ( times , body ) {
	for ( var i = 0; i < times ; i ++) body (i);
}
repeat (3 , function ( n ) {
		unless ( n % 2, function () {
		console . log (n , " is even ") ;
	}) ;
}) ;

The two functions unless and repeat help us to create a new control flow. The working of the above code goes like this: repeat simply repeats a given function as many times as you want and passes in the current count of the iteration to the function. So, any function you pass in can use this value if it likes. In our case, our function makes use of this value. It then calls unless and passes in the value generated by n%2. unless works like “unless this condition is true, you execute the passed function”. In our case, the passed function just logs the value.

Finally, they help you modify other functions:

function make(func,value){
	return func(value);
}
console.log(make(Boolean,0));
console.log(make(Number,"12"));

This one is pretty self-explanatory. Depending on what you pass in as the first argument, you get a corresponding return value.

That’s it on higher order functions.

Simplifying Lexical Scoping and Closures in JavaScript

In this post I will quickly introduce the concepts of lexical scoping and closures in JavaScript. Both of them are quite easy to understand so here we go with an example:

function outer(a){
	function inner(){
		console.log(a);
	}
	inner();
}

In the above snippet, the outer function accepts a parameter a and the function inner logs it to the browser console. This is possible because of lexical scoping. Lexical scoping determines how variable scopes are resolved in inner functions. An inner function(s) has access to the variables of its enclosing function and that is why inner has access to a.

Here is a slightly trickier example:

function outer(a){
	function inner(){
		var a = 5;
		console.log(a);
	}
	inner();
}

This code will always log 5 to the console because the inner function inner has a variable named a which is also the name of the variable of the enclosing function. In this case, the inner a will “hide” the outer a.
That’s it for lexical scoping – how variables are scoped for inner functions.

Onto closures. Consider the snippet:

function greaterThan(m){
	return function(n){ return n > m; }
}
var gt10 = greaterThan(10);
console.log(gt10(32));

The browser console will log true since 32 is greater than 10. What is happening might not be so intuitive at first but a closer look will show that it is not all so hard to grasp. First of all, functions are just regular values – deal with it. Here is how this works:
You create a function named greaterThan that returns another function named.. er.. unnamed actually. Since functions can be stored in variables, you store the returned function in a variable named gt10. This can be used to compare a given value and see if it is greater than 10. Why 10? Because you initially passed in 10 to greaterThan. The inner function “remembers” that you had passed in 10 to its outer function when it was returned. When this returned function is called, it compares the passed argument with the one that was passed to its outer function. The returned function is a closure.

A closure, thus, is an inner function that remembers the environment (values of the outer function) in which it was created. That’s it.

Listing, Adding, and Removing Properties in JavaScript Object

In this post I will show you how to list all the properties of a JavaScript object, their values, and how you can add and remove properties. Let’s jump straight into it.

So we have a person named John Doe and we would like to list down all the properties of the person object along with their values. Here is how to do it:

var person = {name:"John Doe",age:27};
for(var property in person){
	console.log(property + ":" + person[property]);
}

This will also list the global properties (variables), if any, declared on the window object.

This leads to the next question: what if I want to list all the properties that a person has? I dont care for the values, just the properties. Well, use the Object.keys(obj) function. This will return an array containing all the properties of the object.

var person = {name:"John Doe",age:27};
var props = Object.keys(person);
console.log(props);

Now let’s say that John Doe wants to register himself as a voter. To do this, we must first check if he is atleast 18 years old. Then if he is 18, give him voting rights by adding voter : true to his object. Here is how you’d do it:

var person = {name:"John Doe",age:27};
function registerVoter(person){
	if(!"age" in person)
		return;
	// or person.age >= 18
	if(person["age"] >= 18)
		person["voter"] = true;
		// or person.voter = true;
}
registerVoter(person);
console.log(person);

A few years later, John Doe got a rap sheet and his voting right was taken away from him. Here is how you’d take away his voting right:

var person = {name:"John Doe",age:27};
function unregisterVoter(person){
	if("voter" in person)
		// or delete person.voter
		delete person["voter"];
}
unregisterVoter(person);
console.log(person);

That’s it!

How to calculate MD5 hash in Java?

MD5 is a cryptographic hash function that takes in arbirtary data and produces a 128-bit hash from it. This hash can be used to verify the integrity of data. For example, you can check the integrity of the file you received over the internet by calculating its MD5 hash and comparing it with the hash that was advertised. If the two hash values do not match, there is something definitely wrong. Either the file is corrupt or it has been changed in transit. In any case, it is better to just discard the file.

Without further ado, here is the code to produce MD5 hash of the string “hello world”:

import java.security.MessageDigest;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.Path;
import javax.xml.bind.DatatypeConverter;

MessageDigest md = MessageDigest.getInstance("MD5");
byte[] digest = md.digest("hello world".getBytes("UTF-8"));
println "${DatatypeConverter.printHexBinary(digest)}"

The MessageDigest class allows you to generate the message digest using an algorithm like SHA-1, MD5, etc. The digest is generated from bytes of data so we need to call the getBytes(). The digest() accepts the bytes and computes the hash. The computed hash is returned as byte[]. The printHexBinary() method converts the hash in bytes into a string.

The following code shows computing the MD5 hash of a file:

import java.security.MessageDigest;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.Path;
import javax.xml.bind.DatatypeConverter;

Path path = Paths.get( System.getProperty("user.home") ).resolve("2015-03-28.16.18.33_002.png");
byte[] data = Files.readAllBytes(path);
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] digest = md.digest(data);
println "${DatatypeConverter.printHexBinary(digest)}"

The readAllBytes method reads all the bytes of the file into memory and then calculates the hash value. If you do not want to copy all the bytes into memory at once, you can do it as follows:

import java.security.MessageDigest;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.Path;
import javax.xml.bind.DatatypeConverter;

MessageDigest md = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[1024];
int read = 0;
File file = new File(System.getProperty("user.home"),"2015-03-28.16.18.33_002.png");
InputStream is = new FileInputStream(file);
while( (read=is.read(buffer)) != -1){
    md.update(buffer,0,read);
}
byte[] digest = md.digest();
println "${DatatypeConverter.printHexBinary(digest)}"

That’s it on computing MD5 in Java

Deserializing JSON using GSON – a Subtle Difference

Today I was asked to deserialize two similar-looking JSON strings and explain why both of them required slightly different code. This post summarizes that. I deserialized JSON using GSON and the code is written in Groovy. To begin, here is the first JSON string:

[
    {
        "name": "test1",
        "nestedObject": {
            "id": "1",
            "name": "Banana"
        }
    },
    {
        "name": "test2",
        "nestedObject": {
            "id": "2",
            "name": "Apple"
        }
    }
]

and here is the second JSON string:

{
    "elements": [
        {
            "name": "test1",
            "nestedObject": {
                "id": "1",
                "name": "Banana"
            }
        },
        {
            "name": "test2",
            "nestedObject": {
                "id": "2",
                "name": "Apple"
            }
        }
    ]
}

They both provide the same information – an array of elements. However, they are different. The first string contains an array. The second string contains an object which contains an array. Hence, the code required for deserialization will be different. The code required for deserializing the first string is this:

@Grab( 'com.google.code.gson:gson:2.3.1' )
import com.google.gson.Gson;
import com.google.gson.annotations.SerializedName;
import com.google.gson.reflect.TypeToken;
import java.lang.reflect.Type;

class Element{
    String name;
    @SerializedName("nestedObject") NestedObject obj;
    public static class NestedObject{
        String id;
        String name;
    }
}

String json = """
[
    {
        "name": "test1",
        "nestedObject": {
            "id": "1",
            "name": "Banana"
        }
    },
    {
        "name": "test2",
        "nestedObject": {
            "id": "2",
            "name": "Apple"
        }
    }
]
"""

Gson gson = new Gson();
Type collectionType = new TypeToken<List<Element>>(){}.getType();
List<Element> elements = (List<Element>) new Gson().fromJson(json,collectionType);
println elements.size();

An array contains elements so Element seems like an apt name for the class that represents them. If you observe each element of the array in the JSON strings, you will find that they contain a nested object. This calls for making a nested class in the Element class named NestedObject. We are deserializing the JSON string which contains an array of our custom object (with a nested object) and hence we need to get a TypeToken which represents the class of the object. Since our class is named Element, we will need a List type token. Once we are done with that, the rest is just GSON magic. The last two lines deserialize the JSON and print the number of elements in the array (err, List), respectively.

Now moving on to the next JSON string. The code for deserialization is as follows:

@Grab( 'com.google.code.gson:gson:2.3.1' )
import com.google.gson.Gson;
import com.google.gson.annotations.SerializedName;
import com.google.gson.reflect.TypeToken;
import java.lang.reflect.Type;

class Element{
    String name;
    @SerializedName("nestedObject") NestedObject obj;
    public static class NestedObject{
        String id;
        String name;
    }
}

class Response{
    List<Element> elements;
}

String json = """
{
    "elements": [
        {
            "name": "test1",
            "nestedObject": {
                "id": "1",
                "name": "Banana"
            }
        },
        {
            "name": "test2",
            "nestedObject": {
                "id": "2",
                "name": "Apple"
            }
        }
    ]
}
"""

Gson gson = new Gson();
Response r = gson.fromJson(json,Response.class);
println r.elements.size();

This is pretty straight-forward. We have a response, represented by the Response class. The response contains an array of elements and that array is named “elements”. Hence, the List in Response is named elements as well. We no longer need the TypeToken because now we are deserializaing an object rather than an array of objects. It doesn’t matter if the object being deserialized contains an array within it. The “outer most” thing to deserialize is a single object.

That is the difference between the two JSON strings. Until next time.

What is HTTP/2? Part 2

In the last post I mentioned the problems HTTP/1.1 has. In this post I will be talking about Google’s SPDY and why it has been chosen as a starting point for HTTP/2.

HTTP relies on TCP for the transport of packets. HTTP requests tend to be short and bursty while TCP is designed for long-lived connections. Furthermore, TCP has a “slow start” as a part of its congestion control strategy. Congestion control is all good but given the nature of HTTP requests, it can lead to network under-utilization. Opening multiple TCP connections can lead to an unfair use of network resources, depriving others of their fair share. Another restriction is that the requests are exclusively client-initiated meaning that unless the client requests for something, the server cannot send it. All of this dictates that changes be made in how HTTP works.

This is where SPDY (SPeeDY) comes in. SPDY is a session layer protocol that works with SSL to allow many concurrent HTTP requests to run using a single TCP connection. It also aims to overcome the other problems in HTTP/1.1 like verbose headers and so on. Here are the features of SPDY:

Multiplexed Streams

SPDY overcomes the HOL blocking problem in HTTP/1.1 by allowing multiplexed streams. Each request is associated with a uniquely identifiable stream and many such streams exist over a single TCP connection. This means that even if the response for a large resource takes time, the responses for smaller resources can still be sent over their stream. Furthermore, the responses can even be interleaved meaning that a part of the larger resource will be sent followed by a part of the smaller resource (or the smaller resource completely, if it is that small) and so on. This works well with TCP as fewer connections will be made and fewer, densely packed packets will be sent.

Request Prioritization

Suppose you open a webpage. The normal way in which the browser works is that it will load all the HTML first, parse it, and then request for further resources like CSS, JavaScript, and media. These requests are made in the order in which they appear in the HTML; what comes first is requested first. This can be a little problematic if you decide to scroll down while the images for the upper part of the webpage are still being loaded. SPDY solves this problem by allowing stream prioritization. As soon as you scroll down, your browser can tell the server that the stream for loading the new image, which came into view because you scrolled down, is more important than the one at the top. The server can respond accordingly, thus reducing latency.

Header Compression

HTTP headers are very verbose and transmitting the headers itself is a resource hog. SPDY compresses these headers before sending them over the wire. This leads to a saving between 45 to 1142 ms of page load time.

Server Push

Another great feature of SPDY is server-initiated pushes. Traditionally, the server only sends resources in response to requests. There are some resources that will implicitly be needed. For example, JavaScript and CSS. Instead of waiting for the client to send the request, the server could just send the file. This would eliminate the need for the client to make a request and thus reduce latency.

All of these features present in SPDY made IETF spring into action – either Google develops a proprietary replacement for HTTP or a newer version of HTTP needs to be developed in the open. So using SPDY as a starting point, IETF is developing HTTP/2 which is now nearing completion. Google has announced that they will discontinue SPDY in favor of HTTP/2.

There are a few ways in which HTTP2 differs from SPDY.

ALPN

SPDY uses NPN (Next Protocol Mechanism). NPN allows a TLS connection to negotiate which application-level protocol it will be running across (spdy/3, spdy/2 or http/1.1). NPN works on the same level as SSL does – the session layer. ALPN (Application-Layer Protocol Negotiation) will work in the same way as NPN but the negotiation will be done in the application layer.

HPACK

SPDY uses GZIP to compress headers. GZIP is good but it is vulnerable to CRIME (Compression Ratio Info-leak Made Easy). So IETF came up with HPACK which is a new, header-specific compression scheme and is much safer than GZIP.

To conclude, HTTP/2 will make the internet much faster without changing any of the semantics of HTTP/1.1. The reduction in latency and the amount of data to be transmitted is sure to make the web much more accessible in places where a higher bandwidth is still a rarity. For those of us who do have higher bandwidths, HTTP/2 will make the most of it. Looking forward to seeing it in action.

Posted in web

What is HTTP/2? Part 1

HTTP, the world’s most widely used application protocol, has remained unchanged for the last 15 years. However, the World Wide Web that it supports has changed phenomenally. This change has made IETF come up with the second major version of HTTP – the HTTP/2. Before I go into HTTP/2, here is a quick overview of what is wrong with HTTP/1.1:

The Web has Evolved

The modern websites are no longer mere static pages. They are rich and full of CSS, JavaScript, and media. According to HttpArhive, an average webpage has a size of 2MB. Out of these 2MB, images alone take up 1.2MB. It isn’t uncommon to have at least 40 requests just to fetch the images on a webpage. Due to the way HTTP/1.1 is designed, you can only have up to 6 simultaneous connections to a server. This means a lot of time is spent just fetching the images. This severely increases the overall page load time.

To overcome this problem, HTTP pipelining is used. In this strategy, multiple HTTP requests are sent over a single TCP connection. Although this results in drastic improvements, it suffers from “head of line” blocking. Suppose if requests are sent for resources A-B-C-D, pipelining requires that the responses be received in the same order. In other words, it is first-come-first-served. If response to resource A takes a lot of time, the responses to B-C-D will be blocked. This again increases the load time. Such delays can have a severe impact on user engagement which directly affects the revenue generated.

Verbose HTTP Headers

GET / HTTP/1.1
Host: www.etsy.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
DNT: 1
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Cookie: uaid=uaid%3DVdhk5W6sexG-_Y7ZBeQFa3cq7yMQ%26_now%3D1325204464%26_slt%3Ds_LCLVpU%26_kid%3D1%26_ver%3D1%26_mac%3DlVnlM3hMdb3Cs3hqMVuk_dQEixsqQzUlNYCs9H_Kj8c.; user_prefs=1&2596706699&q0tPzMlJLaoEAA==
Connection: keep-alive

This is the header from the first request to load etsy.com and takes 525 bytes.

GET /assets/dist/js/etsy.recent-searches.20121001205006.js HTTP/1.1
Host: www.etsy.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14
Accept: */*
DNT: 1
Referer: http://www.etsy.com/
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Cookie: autosuggest_split=1; etala=111461200.1476767743.1349274889.1349274889.1349274889.1.0; etalb=111461200.1.10.1349274889; last_browse_page=%2F; uaid=uaid%3DVdhk5W6sexG-_Y7ZBeQFa3cq7yMQ%26_now%3D1325204464%26_slt%3Ds_LCLVpU%26_kid%3D1%26_ver%3D1%26_mac%3DlVnlM3hMdb3Cs3hqMVuk_dQEixsqQzUlNYCs9H_Kj8c.; user_prefs=1&2596706699&q0tPzMlJLaoEAA==
Connection: keep-alive

This is the header from the subsequent request that is made to Etsy to load the JavaScript file. The header is 690 bytes in total but only 290 bytes are new. There is a lot of redundant data being sent. For example, the “User-Agent”. Sending redundant data not only takes up bandwidth but also increases latency.

Over the time, there have been a number of “solutions” to these problems but most of them have been just hacks. Here is the list of hacks that the developers have come up with:

Spriting

Instead of making 40 requests to fetch 40 different images, what if you combined them into one single image that contained all the 40 images and then applied CSS to select the right one? This is the idea behind spiriting. Spriting is nifty but even the tiniest of change to the sprite requires invalidating the entire sprite from the client’s cache.

Sharding

HTTP/1.1 limits you to 6 simultaneous connections to a server. If you spread your resources across multiple servers, you can make more connections and fetch the resources faster. This can be a major performance win but it goes against the concept of “fairness”. If on a network the web browser opens up 40 different connections to 7 different servers to fetch the images, it will choke the network and prevent other applications, like VoIP, from getting their fair share of resources.

Concatenation

Concatenation refers to combining multiple CSS or JavaScript files to reduce the number of requests that will be made to fetch them. Concatenation is used in conjunction with minification. Minification removes all the whitespaces and supplementary content, like comments and new line characters, etc., from the files to reduce their size.

Inlining

Suppose that there is an img tag in the HTML. The src of that image specifies the location of that image. Generally, the HTML does not specify a server. This means that the browser looks for the image on the same server that served the HTML. If, however, the server is specified, the browser will contact that server and fetch the image from there. This overcomes the 6 connection constraint inherent to HTTP/1.1

To genuinely overcome these problems, Google developed an open networking protocol SPDY (pronounced “speedy”). SPDY and HTTP/2 will be the topic of my next post.

Installing the Android SDK on Ubuntu 14.04

Setting up the Android SDK on Ubuntu is slightly more involved than doing the same on Windows. This step-by-step guide is aimed at making it easy for you to set up the SDK on Ubuntu.

Enable Virtualization Support from BIOS

Enable virtualization from the BIOS. This is required for the Android AVD to start.

Download the Android SDK

Go to http://developer.android.com/sdk/index.html and scroll down till you find the “Other Download Options” section. In the “SDK Tools Only” section, select the Linux package. Wait for the download to finish.

Extracting the SDK

By default, the downloads go to the Downloads folder in the home directory. We will, however, move the SDK archive to /usr/local and extract it there. Fire up a terminal window by pressing CTRL - ALT - T. Then type the following:

cd ~/Downloads
sudo mv android* /usr/local
cd /usr/local
sudo tar zxvf android*

Using ls will show you the android-sdk-linux folder.

Changing the Permissions

You also need to change the permissions for the android-sdk-linux. In the terminal window, type the following:

sudo chmod -R ugo+rwx android-sdk-linux

This will recursively give read, write, and execute permissions to the android-sdk-linux and all it’s children. This can take a few seconds to complete.

Setting the Environment Variables

You now need to set the environment variables. In your terminal window type the following:

sudo gedit /etc/profile

and append the following:

  
export PATH=${PATH}:/usr/local/android/android-sdk-linux/tools
export PATH=${PATH}:/usr/local/android/android-sdk-linux/platform-tools

Save the file. Now you need to reload these variables. In your terminal, type:

. /etc/profile

For the most part, you are done. There are a few additional libraries you need if you are on a 64-bit machine. If you are, type the following in your terminal:

sudo apt-get install lib32stdc++6
sudo apt-get install lib32z1

These are the 32-bit libraries that the Android SDK needs.

Checking the Setup

Time to check if all went well. Type the following in your terminal:

cd ~
android

This should fire up your Android SDK Manager.

android avd

This should fire up the Android AVD Manager.

adb version

This should display the ADB version.

That’s about it for setting up the Android SDK on Ubuntu.