How to Set Up an Elasticsearch Cluster on Ubuntu 18.04

July 13, 2021

by Thomas Tran

This article serves as a guide on how to install an Elasticsearch cluster on three Ubuntu 18.04 machines. There are ways to automatically deploy an Elasticsearch cluster using Kubernetes or Docker Cluster, but this guide will serve as a guide on how to manually set up a cluster.

But first a few terminologies:

Cluster - a collection of machines/servers that all have Elasticsearch installed, delegate responsibilities amongst themselves, and that all communicate with each other through a collective API.
Nodes - a machine/server in the cluster

First, SSH into all ES nodes in the cluster:

ssh <ubuntu username>@<ES node ip>

Elasticsearch can’t be run as a root user. Elasticsearch itself restricts this. We need to create a new user named elasticsearch.

sudo adduser elasticsearch
sudo adduser elasticsearch sudo
sudo -su elasticsearch

After you log in, let’s install some prerequisites:

sudo apt-get install unzip

sudo apt install openjdk-11-jre-headless

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

Now let’s install Elasticsearch. In this article, we’ll install version 7.13.3, but you can install any other versions that you want.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get update && sudo apt-get install elasticsearch=7.13.3

Add custom settings to `elasticsearch.yml`

Now let’s add custom settings to elasticsearch.yml. This YAML file is used to configure Elasticsearch.

Let’s have one of our machines be a “master node”, which is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. It is important for cluster health to have a stable master node.

Do this command only on the master node in the cluster:

sudo bash -c 'cat > /etc/elasticsearch/elasticsearch.yml << EOF
cluster.name: cool-cluster

node.name: ${HOSTNAME} # This will default to the host name of our machine
node.master: true
node.data: true

path.logs: /var/log/elasticsearch
path.data: /usr/share/elasticsearch/data

bootstrap.memory_lock: true

network.host: <NODE IP ADDRESS>
discovery.seed_hosts: ["<NODE IP ADDRESS MASTER NODE>", "<NODE IP ADDRESS DATA NODE 1>", "<NODE IP ADDRESS DATA NODE 2>"]
cluster.initial_master_nodes: ["<NODE IP ADDRESS MASTER NODE>"]

xpack.security.enabled: true

# This turns on TLS/SSL for inter-node communication
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12

# This turns on TLS/SSL for the HTTP (Rest) interface
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elasticsearch-ssl-http/elasticsearch/${HOSTNAME}/http.p12

# These settings need to be entered in elasticsearch-keystore
# xpack.security.transport.ssl.keystore.secure_password
# xpack.security.transport.ssl.truststore.secure_password
# xpack.security.http.ssl.keystore.secure_password
EOF'

Now configure elasticsearch.yml for each data node in the cluster. Repeat for all data nodes:

sudo bash -c 'cat > /etc/elasticsearch/elasticsearch.yml << EOF
cluster.name: cool-cluster

node.name: ${HOSTNAME}
node.master: false
node.data: true

path.logs: /var/log/elasticsearch
path.data: /usr/share/elasticsearch/data
path.home: /usr/share/elasticsearch

bootstrap.memory_lock: true

network.host: <PRIVATE IP ADDRESS>
discovery.seed_hosts: ["<PRIVATE IP ADDRESS MASTER NODE>", "<PRIVATE IP ADDRESS DATA NODE 1>", "<PRIVATE IP ADDRESS DATA NODE 2>"]
cluster.initial_master_nodes: ["<PRIVATE IP ADDRESS MASTER NODE>"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

# This turns on TLS/SSL for inter-node communication
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12

# This turns on SSL for the HTTP (Rest) interface
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elasticsearch-ssl-http/elasticsearch/${HOSTNAME}/http.p12

# These settings need to be entered in elasticsearch-keystore
# xpack.security.transport.ssl.keystore.secure_password
# xpack.security.transport.ssl.truststore.secure_password
# xpack.security.http.ssl.keystore.secure_password
EOF'

Open ports 9200-9300

Now that you’ve configured elasticsearch.yml for each node in the cluster, we need to open ports 9200-9300 so that clients can interact with Elasticsearch and nodes can interact with each other.

Repeat this for every node in the cluster:

sudo ufw allow 9200:9300/tcp
sudo ufw allow ssh
sudo ufw enable

Increase file descriptors

Now let’s increase file descriptors on each node. Elasticsearch recommends doing this because it uses a lot of file descriptors or file handles. Running out of file descriptors can be disastrous and will most probably lead to data loss. Make sure to increase the limit on the number of open files descriptors for the user running Elasticsearch to 65536 or higher.

sudo bash -c 'cat > /etc/default/elasticsearch << EOF
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65536
MAX_LOCKED_MEMORY=unlimited
EOF'

Disable Swapping

Next, let’s disable swapping. Swapping is very bad for performance, for node stability, and should be avoided at all costs. It can cause garbage collections to last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster. In a resilient distributed system, it’s more effective to let the operating system kill the node.

First, add LimitMEMLOCK=infinity to /usr/lib/systemd/system/elasticsearch.service

sudo vim /usr/lib/systemd/system/elasticsearch.service

You’ll have to manually add LimitMEMLOCK=infinity under the [Service] section:

sudo bash -c 'cat > /etc/security/limits.conf << EOF
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
EOF'

Increase `vm.max_map_count`

Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count.

sudo vim /etc/sysctl.conf

Add vm.max_map_count=262144 manually to the bottom of the file. Verify your change with:

sudo sysctl -p

Change the permission of the ES folder

sudo chown -R elasticsearch:elasticsearch /usr/share/elasticsearch

If you don’t do this, then the elasticsearch user will not be able to access this folder when running.

Enable TLS/SSL Inter-node communications

Before you enable TLS/SSL, make sure that you stop Elasticsearch and Kibana on all nodes in the cluster:

sudo service elasticsearch stop

sudo service kibana stop

Do for master node only:

sudo ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-certutil ca

sudo ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /usr/share/elasticsearch/elastic-stack-ca.p12

sudo ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-certutil http

Do for each node:

sudo mkdir /etc/elasticsearch/certs
sudo mkdir /etc/elasticsearch/certs/elasticsearch-ssl-http

sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch /etc/elasticsearch/certs

Do on master node:

sudo cp /usr/share/elasticsearch/elastic-certificates.p12 /etc/elasticsearch/certs

sudo cp /usr/share/elasticsearch/elastic-stack-ca.p12 /etc/elasticsearch/certs

sudo cp /usr/share/elasticsearch/elasticsearch-ssl-http.zip /etc/elasticsearch/certs

Copy cert from master node to other nodes in the cluster

sudo scp /etc/elasticsearch/certs/elastic-certificates.p12 elasticsearch@<REMOTE PRIVATE IP>:/etc/elasticsearch/certs/

sudo scp /etc/elasticsearch/certs/elastic-stack-ca.p12 elasticsearch@<REMOTE PRIVATE IP>:/etc/elasticsearch/certs/

sudo scp /etc/elasticsearch/certs/elasticsearch-ssl-http.zip elasticsearch@<REMOTE PRIVATE IP>:/etc/elasticsearch/certs/

Do for each node:

sudo chmod -R elasticsearch:elasticsearch /etc/elasticsearch/certs/*

sudo unzip /etc/elasticsearch/certs/elasticsearch-ssl-http.zip -d /etc/elasticsearch/certs/elasticsearch-ssl-http

For each environment, you’ll have to add

xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
xpack.security.http.ssl.keystore.secure_password

if you decide to generate a TLS/SSL certificate with a password.

ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

It will prompt you for the value of the setting you want to insert. Enter the password to your TLS/SSL certificates.

ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

It will prompt you for the value of the setting you want to insert. Enter the password to your TLS/SSL Central Authority (CA).

ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

Repeat this for each node.

Do this from a terminal on your local computer:

This downloads the Certificate Authority (CA) and certificate from the remote server so that we can back it up in case things go bad.

scp elasticsearch@<REMOTE PRIVATE IP>:/etc/elasticsearch/certs/elastic-stack-ca.p12 .

scp elasticsearch@<REMOTE PRIVATE IP>:/etc/elasticsearch/certs/elastic-certificates.p12 .

scp elasticsearch@<REMOTE PRIVATE IP>:/etc/elasticsearch/certs/elasticsearch-ssl-http.zip .

Tip: remember the period (.) at the end!

Check certificate expiry date

Note that we’ll have to renew the certificate and Certificate Authority every 3 years.

openssl pkcs12 -in elastic-certificates.p12 -nodes | openssl x509 -noout -enddate

Generate user credentials

This is a one-time operation that will generate all credentials needed to access Elasticsearch API, Kibana, etc… Be careful when you’re running this command. ES will not let you run it a second time! If you mess it up, you’ll have to reinstall ES, and delete the etc/elasticsearch, /usr/share/elasticsearch folders!

sudo ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

Start Elasticsearch

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Test things out

curl -XGET <http://<master node ip>/
curl -XGET <http://<master node ip>:9200/_cluster/health?pretty

Check logs

sudo tail /var/log/elasticsearch/<cluster name>.log

How to Set Up an Elasticsearch Cluster on Ubuntu 18.04

Add custom settings to elasticsearch.yml

Open ports 9200-9300

Increase file descriptors

Disable Swapping

Increase vm.max_map_count

Change the permission of the ES folder

Enable TLS/SSL Inter-node communications

Do for master node only:

Do for each node:

Do on master node:

Copy cert from master node to other nodes in the cluster

Do for each node:

Do this from a terminal on your local computer:

Check certificate expiry date

Generate user credentials

Start Elasticsearch

Test things out

Check logs

Add custom settings to `elasticsearch.yml`

Increase `vm.max_map_count`