Elasticsearch and NodeJS Tutorial

Elasticsearch is a RESTful search and analytics database. I have worked with Elasticsearch in more than one projects but every time I do I end up looking for the same things online. The documentation is not very well organized and it’s difficult to quickly find the things needed.

Hence I have decided to come up with this Elasticsearch NodeJS tutorial for beginners. This tutorial will save a lot of your time wasted in searching for stuff and get you up to speed with the most important basic queries of Elasticsearch. Reference links are also provided wherever applicable.

This tutorial assumes that you have Elasticsearch installed and running on your local machine. Also, you must have node.js installed

We are using…

• Elasticsearch v6.6.1
• NodeJS v8.9.3

Through this article, we will follow a simple pattern. We will create separate files for different functionalities, within which we will export a function for that functionality and below it we will see how we can invoke the function.

Connecting to ElasticSearch using Node.js

Create a new npm project.

mkdir nodejs-elasticsearch
cd nodejs-elasticsearch
npm init --yes

Now install the npm package elasticsearch, this package provides a wrapper around elasticsearch service.

npm i elasticsearch--save

Now let’s create a new file that will be our client. We will call this file client.js.

const es = require('elasticsearch');
const esClient = new es.Client({
    host: 'localhost:9200',
    log: 'trace'
});

module.exports = esClient;

Now let’s create a new file that will basically check if we have successfully connected to elasticsearch service by sending a ping. Let’s call this file ping.js.

const esClient = require('./client');

esClient.ping({
// ping usually has a 3000ms timeout
    requestTimeout: 1000
}, function (error) {
    if (error) {
        console.trace('elasticsearch cluster is down!');
    } else {
        console.log('All is well');
    }
});

Now let’s test out our connection. Run…

node ping.js

//output
Elasticsearch INFO: 2019-03-11T12:09:31Z
  Adding connection to http://localhost:9200/

Elasticsearch DEBUG: 2019-03-11T12:09:31Z
  starting request {
    "method": "HEAD",
    "requestTimeout": 1000,
    "castExists": true,
    "path": "/",
    "query": {}
  }


Elasticsearch TRACE: 2019-03-11T12:09:31Z
  -> HEAD http://localhost:9200/

  <- 200


Elasticsearch DEBUG: 2019-03-11T12:09:31Z
  Request complete

All is well

Creating Index in Elasticsearch

Index in Elasticsearch are analogous to database in other storage engines( eg: MongoDB).

So to create an index let’s create another file named createIndex.js.

const esClient = require('./client');
const createIndex = async function(indexName){
    return await esClient.indices.create({
        index: indexName
    });
}

module.exports = createIndex;

A simple function that takes an index. To test this let’s add a test function that will create an index named blog.

const esClient = require('./client');
const createIndex = async function(indexName){
    return await esClient.indices.create({
        index: indexName
    });
}

module.exports = createIndex;


async function test(){
    try {
        const resp = await createIndex('blog');
        console.log(resp);
    } catch (e) {
        console.log(e);
    }
}
test();

Now let’s test it out.

node createIndex.js

//console output
Elasticsearch INFO: 2019-03-11T12:25:20Z
  Adding connection to http://localhost:9200/

Elasticsearch DEBUG: 2019-03-11T12:25:20Z
  starting request {
    "method": "PUT",
    "path": "/blog",
    "query": {}
  }


Elasticsearch TRACE: 2019-03-11T12:25:23Z
  -> PUT http://localhost:9200/blog

  <- 200
  {
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "blog"
  }

Elasticsearch DEBUG: 2019-03-11T12:25:23Z
  Request complete

{ acknowledged: true, shards_acknowledged: true, index: 'blog' }

This has created an index named blog for us.

Creating mapping in ElasticSearch

Mapping is like defining a structure to your documents, it specifies how your documents will be stored and indexed.

To create a mapping we use the putMapping method, as shown in the example below.

const esClient = require('./client');
const addmappingToIndex = async function(indexName, mappingType, mapping){
    console.log(mapping);
    return await esClient.indices.putMapping({
        index: indexName,
        type: mappingType,
        body: mapping
    });
}

module.exports = addmappingToIndex;


// test function to explain how to invoke.
async function test(){
    const mapping = {
        properties: {
            title: {
                type: "text"
            },
            tags: {
                type: "keyword"
            },
            body: {
                type: "text"
            },
            timestamp: {
                type: "date",
                format: "epoch_millis"
            }
        }
    }
    try {
        const resp = await addmappingToIndex('blog', 'ciphertrick', mapping);
        console.log(resp);
    } catch (e) {
        console.log(e);
    }
}


//test();

Text is the most common type of field type used, Keyword is used if you want to perform aggregated operations. There are more types which you can read about here.

Indexing Data into Elasticsearch

Indexing Data basically means inserting data into Elasticsearch. Let’s create a file named insertData.js.

const esClient = require('./client');
const insertDoc = async function(indexName, _id, mappingType, data){
    return await esClient.index({
        index: indexName,
        type: mappingType,
        id: _id,
        body: data
    });
}

module.exports = insertDoc;


async function test(){
    const data = {
        title: "Learn elastic search",
        tags: ['NodeJS', 'Programming'],
        body: `Lot of content here...
                .... article`
    }
    try {
        const resp = await insertDoc('blog', 2, 'ciphertrick', data);
        console.log(resp);
    } catch (e) {
        console.log(e);
    }
}


//test();

Here is index function takes an object with index(name of the index), type(name of mapping), id(this is the unique identifier of the object), body(actual data to be indexed).

Performing Simple search in Elasticsearch

const esClient = require('./client');
const searchDoc = async function(indexName, mappingType, payload){
    return await esClient.search({
        index: indexName,
        type: mappingType,
        body: payload
    });
}

module.exports = searchDoc;


async function test(){
    const body = {
        query: {
            match: {
                "title": "Learn"
            }
        }
    }
    try {
        const resp = await searchDoc('blog', 'ciphertrick', body);
        console.log(resp);
    } catch (e) {
        console.log(e);
    }
}


//test();

// More details here https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html

This particular search will return items in which the title has the word ‘Learn‘.

eg: Query
    const body = {
        query: {
            match: {
                "title": "Learn"
            }
        }
    }

Search as you type in ElasticSearch

Elastic search has various variations of search, in the prior example the search will match the entire word passed in the query, so articles having ‘Learn’ word in the title will be hit. But if you try changing that to `L` or ‘Lear’, it wouldn’t give you any result.

To solve this type of query we have match_phrase_prefix. This is helpful in returning search results as you type your search query. The way you might have seen while trying to search for something on Google.

const esClient = require('./client');
const searchDoc = async function(indexName, mappingType, payload){
    return await esClient.search({
        index: indexName,
        type: mappingType,
        body: payload
    });
}

module.exports = searchDoc;


async function test(){
    const body = {
        query: {
            match_phrase_prefix: {
                "title": "Lea"
            }
        }
    }
    try {
        const resp = await searchDoc('blog', 'ciphertrick', body);
        console.log(resp);
    } catch (e) {
        console.log(e);
    }
}


//test();

// More details here https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html

There are more options available in full-text search. You can browse them on this link.

Aggregation and faceted search in Elasticsearch

This is one of the most important type of search available on Elasticsearch.

If you have used amazon for buying something chances are that you have noticed searches which goes as follows…

“keyboards in Computer Accessories“, “knife in Kitchen Appliances”

You would have also seen specifications shown along with the count on the left menu. eg: Windows (10), Ubuntu (4).

These type of results can be achieved by using Elasticsearch’s Aggregated search also known as Faceted search

Let’s checkout the code to see how we can achieve this in Elasticsearch.

const esClient = require('./client');
const searchDoc = async function(indexName, mappingType, payload){
    return await esClient.search({
        index: indexName,
        type: mappingType,
        body: payload
    });
}

module.exports = searchDoc;


async function test(){
    const body = {
        query: {
            match: {
                "title": "Learn"
            }
        },
        aggs: {
            tags: {
                terms: {
                    field: 'tags'
                }
            }
        }
    }
    try {
        const resp = await searchDoc('blog', 'ciphertrick', body);
        console.log(JSON.stringify(resp));
    } catch (e) {
        console.log(e);
    }
}


//test();

// More details here https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html

Above if you see we are performing aggregation on tags field. This is how the result appears…

{
   "took":8,
   "timed_out":false,
   "_shards":{
      "total":5,
      "successful":5,
      "skipped":0,
      "failed":0
   },
   "hits":{
      "total":2,
      "max_score":0.2876821,
      "hits":[
         {
            "_index":"blog",
            "_type":"test",
            "_id":"2",
            "_score":0.2876821,
            "_source":{
               "title":"Learn elastic search",
               "tags":[
                  "NodeJS",
                  "Programming"
               ],
               "body":"Lot of content here...\n                .... article"
            }
         },
         {
            "_index":"blog",
            "_type":"test",
            "_id":"1",
            "_score":0.2876821,
            "_source":{
               "title":"Learn SQL search",
               "tags":[
                  "SQL",
                  "Programming"
               ],
               "body":"Lot of content here...\n                .... article"
            }
         }
      ]
   },
   "aggregations":{
      "tags":{
         "doc_count_error_upper_bound":0,
         "sum_other_doc_count":0,
         "buckets":[
            {
               "key":"Programming",
               "doc_count":2
            },
            {
               "key":"NodeJS",
               "doc_count":1
            },
            {
               "key":"SQL",
               "doc_count":1
            }
         ]
      }
   }
}

Check out the buckets array within the aggregations property. It returns all the tags which include the article matching the search query, it also returns the document count.

Code

You can find the code in this Github repo.

Conclusion

This was a very comprehensive beginner’s guide for Elastic Search and Node.js. By now you would be able to perform basic operations in elasticesearch. Bookmarking this link would be a good idea, as you will need this in the future whenever you plan to use ElasticSearch.

Also Read…

• Node MySQL Tutorial
• MongoDB and NodeJS beginner’s guide
• Node.js and Redis tutorial