Elasticsearch .NET (using NEST)

If you’re developing a huge e-commerce site or any complex platform, then you will be dealing with huge amounts of data. One of the most common challenges of an undertaking such as that is the efficient storage, fetching, and overall management of the said Big Data. And this is precisely what Elasticsearch aims to address.

What is this tech, you ask? And how exactly does one go about using it with NEST and why should you, in the first place? Before we go into the nitty-gritty, let’s get acquainted with them first, shall we?

Chapter One: Introduction

Elasticsearch and Kibana

Elasticsearch (ES) is an open-source distributed, RESTful search and analytics engine that allows you to search and explore millions of data in a matter of seconds. It uses JavaScript Object Notation (JSON) for formatting documents as well as Java APIs. Elasticsearch can technically store and manage data, but it should NOT BE USED as a database, especially as your primary source of data. Why? Because it can get corrupted (for any number of reasons), and that would spell the beginning of the end for your application or platform. And you wouldn't want that, do you? Besides, the main function of ES is not for database purposes anyway, so there's that.

Elasticsearch is a web server built on Lucene, which is a full-text search engine library written in Java and licensed by Apache. Lucene is the one who is actually doing the heavy lifting, efficiently performing searches and analytics underneath. On top of that, Elasticsearch is also the pivotal component to a set of open-source tools called the Elastic Stack, which delivers enhanced search and analysis capabilities such as data ingestion, storage, analysis, enrichment, and visualization.

But how does Elasticsearch enhance your software or platform exactly? Well, how about we simplify things and give you a real-life scenario.
In Elasticsearch, when you insert data into the database, you need to store the data first into an Index(Table/Database). This data inside the index is called a Document, and it’s written in JSON format. The data from the index is then broken down into chunks called Shards or Tokens. Shards/Tokens are pieces of data from an index that are used by Lucene to perform searches. They are stored into one or more servers, which are called Nodes, and a group of nodes is called a Cluster.

Elasticsearch has a lot of ways of storing and understanding the data that is being indexed. One example of this is by using analyzers. An analyzer is a tool that combines three functions (character filters, tokenizer, and token filters) into one package. You can configure this either in NEST or in Kibana.

How does it actually work? Well, for instance, when you type in the keyword Ph, records that exist with the exact keyword, as well as those with the term Philippines, should display.

To better understand the scenario, below is a visualization of Elasticsearch.

As for the technologies utilized, here is the list :

.NET Core
C#
Angular (front-end)
Java (Lucene)
Elasticsearch 7.5.1(latest)
Kibana 7.5.1(latest)
MySql(database)

Introduction to NEST:

Now, the beauty of ES is its agility when used with other technologies. For instance, ES works very well with NEST, which is a high-level client that maps all the requests and responses as types. Basically, it’s like a framework that translates from Elasticsearch to C#. NEST comes with a strongly typed query DSL (domain-specific language). It also provides high-level abstractions like auto-mapping.

Chapter Two: Installation

How to install Java

Since Elasticsearch is written in Java, then you need to have this installed first before you can use ES.

● Install the latest Java SE Development Kit version, which you can download from the link.

● Open the installation path:

e.g. C:\Program Files\Java\jdk-14.0.1

● Lastly, add Java in your environment variables by following the instructions below:

Open file explorer and right-click on This PC (or your computer’s name). Then, click on Properties.

2. Next, click on Advanced system settings.

3. Under the Advanced tab, click the Environment Variables at the bottom.

4. On the next display, click New…, add the variable name JAVA_HOME, and make sure the variable value should be the path of the installed Java folder.

e.g. C:\Program Files\Java\jdk-14.0.1

Chapter Three: Usage & Initialization

Basically, there are two ways of installing and using ES. One way is by installing it through the .msi installer and the other one is via the .zip file.

Elasticsearch Installation on Windows

In this tutorial, we will install Elasticsearch using the new one. Unlike in the old way, we have to download and install the Java Development Kit(JDK) and add System Environment Variables for Java. As of this writing, the latest Elasticsearch .msi installer can be downloaded here.

Important note:

Change the size from 2gb to 1024mb on installation because of memory issues.

1. Download and Install the .msi installer. Don't do custom installation, yet, because we can add more customization later on. The installation may fail at the end (the MSI installer is still in beta version) but don’t worry about it. Just click Next until you click Finish. For now, we only need the basic installation.

In this installation, Elasticsearch will be installed as a service, and will automatically run after the installation and during the startup.

To check if Elasticsearch is running, open this in the browser:
http://localhost:9200 and you should be able to see something like this.


    ```
        {
                      "name" : "DESKTOP-N115968",
                      "cluster_name" : "elasticsearch",
                      "cluster_uuid" : "jNmkpykMQKO-KvVQUeC1aw",
                      "version" : {
                        "number" : "7.5.1",
                        "build_flavor" : "unknown",
                        "build_type" : "unknown",
                        "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
                        "build_date" : "2019-12-16T22:57:37.835892Z",
                        "build_snapshot" : false,
                        "lucene_version" : "8.3.0",
                        "minimum_wire_compatibility_version" : "6.8.0",
                        "minimum_index_compatibility_version" : "6.0.0-beta1"
                      },
                      "tagline" : "You Know, for Search"
        }
    ```

2. Download Kibana here.

Kibana is your window to the Elastic Stack, where you can visualize and manage your data.

Once downloaded, extract Kibana files and place them to your desired folder/location. For this tutorial, place it in drive C.
Open C:\kibana-7.5.1-windows-x86_64\bin\kibana.bat
You can add the Kibana shortcut on your desktop for easier access, when running Kibana in the future.
Open the Kibana web portal (http://localhost:5601) to check if it's running.
When the server restarts, Kibana won't start automatically and you have to run the kibana.bat manually. Next, we will set Kibana as a service so that it will run automatically during startup.
Download NSSM (Non-Sucking Service Manager) at https://nssm.cc/download and extract the nssm.exe then place it in drive C:
Open Command Prompt as administrator and type C:\nssm.exe install KibanaService. Then, press enter.
A NSSM service installer window will appear. In the application's path, locate the kibana.bat directory. Then click Install Service.
After installing Kibana as a service, run the service using command prompt and type C:\nssm.exe start KibanaService. It should then automatically start when the server restarts.

Configuring Elasticsearch and Kibana in a dedicated Server

If you don't have another machine, you can use a virtual machine as your dedicated server for Elasticsearch. You can use Windows Server or Windows 10. In this tutorial, I used Windows 10 in a HyperV (built-in Virtualization Application in Windows 10 Pro).

1. Install Elasticsearch and download Kibana in the dedicated server by following the steps above.

2. In the above installation of Elasticsearch we used the default directory. Go to the directory below and open the yml file.

C:\ProgramData\Elastic\Elasticsearch\config\elasticsearch.yml

At the bottom part, add the lines below:


      network.host: 0.0.0.0
      cluster.initial_master_nodes: node-1

3. Next, go to Task Manager > Services > and look for Elasticsearch service. Right-click and then click Restart to apply the changes.

4. Open Kibana's Directory. Based on the example above, we placed Kibana files in the drive C.

Open C:\kibana-7.5.1-windows-x86_64\config\kibana.yml

At the bottom part, add the following line:


       server.host: "0.0.0.0"

5. If Kibana is already running, close the console and run it again to apply the changes.

6. If your Windows Defender Firewall is ON. You will not be able to access Elasticsearch and Kibana from a remote client.

You can turn off the firewall to allow Elasticsearch(9200), Kibana(5601), and other ports access but it's not recommended. Turning off the firewall is considered bad practice.

To allow 9200 and 5601 ports in the Firewall, do the following:.

Open Windows Defender Firewall. Then, click Advanced Settings.
Right-click Inbound Rules and click New Rule.
Click Port and then, click Next.
In the Specific Local Ports field, type 9200,5601 and click Next>Next>Next.
Then, type the name of the rule: 'Elasticsearch and Kibana ports'.
Click Finish.

7. Check Elasticsearch and Kibana by opening a browser and browse:


        http://<IPAddressOfTheServerOrPC>:9200 
        http://<IPAddressOfTheServerOrPC>:5601

If you want a more detailed guide on this installation, you can find it here.

Elasticsearch and Kibana Installation via .zip

To initiate this installation, you first need to download the appropriate ES .zip file for your computer.

1. After downloading Elasticsearch and Kibana, put them in one folder.

2. Configure the Elasticsearch first. Go to the root ES folder and open config/elasticsearch.yml. Uncomment (remove hashtag) and change the cluster.name into your own desired cluster name and change the node.name into your own (Advice: Use best practice naming convention such as node01.)

Also, add http.cors to allow requests. See the image below as reference.

3. To start Elasticsearch, open a command prompt and navigate to the root folder of Elasticsearch.

After navigating to the folder, type in .\bin\elasticsearch in the command prompt and press enter. That should trigger the elasticsearch file and it should start accordingly.

4. To start Kibana, open a command prompt and navigate into the root folder of Kibana.

After navigating to the folder, type in .\bin\kibana.bat in the command prompt and press enter. That should trigger the Kibana bat file and it should start accordingly.

After starting Elasticsearch and Kibana, you can now open your browser.

The default port of Elasticsearch is 9200 and you can access it through postman or http://localhost:9200/.
For Kibana you can access it on http://localhost:5601/.

Congratulations! Now that everything is up and running, you are ready to take it for a test drive and start your sample project. Proceed to Part 2 for the continuation of the tutorial.

Elasticsearch .NET (using NEST) - Part 1/3