Adding a second OpenSearch node: what you get, and what you don't

I have two servers and one OpenSearch node. The obvious move is to put a node on the second server and call it a cluster.

That works, and it is worth doing. But it does not give you high availability, and the reason is counterintuitive enough that a lot of people find out during their first outage. So this article covers the setup and the limitation together.

Why two nodes is not a highly available cluster

OpenSearch elects one node as cluster manager. That node maintains the cluster state: which indexes exist, where the shards live, which nodes are members. To elect a manager or change the state, a majority of the cluster-manager-eligible nodes must agree.

The majority is (n / 2) + 1, where n is the number of cluster-manager-eligible nodes.

Look at the second row. With two eligible nodes the quorum is two, so both must be present. Losing either one leaves you without a majority, and the cluster can no longer elect a manager or update its state.

You have doubled your hardware and your failure surface without gaining any tolerance for failure. Two nodes is not better than one for availability. In one specific sense it is worse, because now there are two machines that can take the cluster down instead of one.

This rule exists to prevent split-brain. If a network problem separated two nodes and each was allowed to carry on as manager, both would accept writes and the two cluster states would diverge. Requiring a majority makes that impossible, because at most one side of a partition can hold a majority.

OpenSearch does adjust the voting configuration automatically as nodes join and leave. Do not plan around that saving you. Design for the quorum arithmetic above and treat any automatic adjustment as a bonus.

So what does the second node actually buy you?

Quite a lot, as long as you are clear that availability is not on the list.

Data redundancy. With one replica per index, every shard exists on both machines. Lose a disk and you have not lost data. This is the main reason to do it.

Read throughput. Search requests can be served by either the primary or the replica shard, so you roughly double read capacity.

Somewhere to fail over to, manually. If a node dies you are down, but the data is intact on the survivor and you can bring the cluster back by hand. That is a much better position than restoring from a snapshot.

A real cluster to test against. Shard allocation, replica behaviour and rolling restarts all behave differently with more than one node. Finding that out on two servers is cheaper than finding it out later.

What it does not buy you is a cluster that survives one machine going away. For that you need a third vote.

The cheap fix: a third node that stores nothing

The third vote does not need to be a third server with matching specs. A cluster-manager-only node holds no data and serves no searches. It participates in elections and holds cluster state, which is a small job.

A small VM is enough. If you have anywhere else to run a container, that is your quorum problem solved, and it turns the second row of that table into the third row.

# node 3, cluster manager only
node.name: search-cm-01
node.roles: [ cluster_manager ]

If you genuinely only have two machines, run two data nodes and accept that you have redundancy but not availability. Just make the decision deliberately, and write it down somewhere, so nobody later assumes the cluster is more resilient than it is.

Setting it up

What follows assumes Ubuntu, OpenSearch 2.x installed from packages, and an existing single node with data on it.

Before you touch anything: snapshot

Register a repository and take a snapshot. The reconfiguration below changes how the existing node forms a cluster, and while it should be safe, "should be" is not a backup.

PUT /_snapshot/backup
{
  "type": "fs",
  "settings": { "location": "/var/backups/opensearch" }
}

PUT /_snapshot/backup/before-clustering?wait_for_completion=true

The path must be listed in path.repo in opensearch.yml and must be writable by the opensearch user.

The single-node trap

Check the existing node's config for this line:

discovery.type: single-node

If it is there, it must go. That setting tells OpenSearch to skip discovery and bootstrapping entirely and elect itself. A node running in that mode will not form a cluster with anyone, and a second node pointed at it will simply never join.

This is the step I would test on a throwaway VM first if the data matters. The node has existing cluster state, so it should re-form as a normal single-node cluster after the setting is removed and the service restarted. Verify that it comes back healthy before you introduce the second node.

Node 1

cluster.name: linkhub-search
node.name: search-01
node.roles: [ cluster_manager, data, ingest ]

network.host: 10.0.0.11
http.port: 9200
transport.port: 9300

discovery.seed_hosts: ["10.0.0.11", "10.0.0.12"]
cluster.initial_cluster_manager_nodes: ["search-01", "search-02"]

path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

Node 2

cluster.name: linkhub-search
node.name: search-02
node.roles: [ cluster_manager, data, ingest ]

network.host: 10.0.0.12
http.port: 9200
transport.port: 9300

discovery.seed_hosts: ["10.0.0.11", "10.0.0.12"]
cluster.initial_cluster_manager_nodes: ["search-01", "search-02"]

path.data: /var/lib/opensearch
path.logs: /var/log/opensearch

Four things about these files.

cluster.name must match exactly. A mismatch is the most common reason a node silently refuses to join. It will start up perfectly happily and form its own cluster of one.

cluster.initial_cluster_manager_nodes must be identical on every node, and it contains node names, not addresses. Different lists on different nodes can produce two separate clusters.

That setting only applies to the very first bootstrap. Once the cluster has formed, it is ignored, and nodes that have already joined store what they need. Leaving it in place is harmless but it is not doing anything. Any node you add later does not need it.

Bind to a private address, not 0.0.0.0. Port 9300 is the transport port and it should never be reachable from the internet. Use your private network, and add a firewall rule allowing 9200 and 9300 only between the two hosts.

sudo ufw allow from 10.0.0.12 to any port 9200,9300 proto tcp

Transport TLS, which is not optional

This is where most two-node setups stall.

With the security plugin enabled, TLS is mandatory on the transport layer. It is optional for the REST layer and required for node-to-node traffic. A single node never exercises this, so it is entirely possible to have run OpenSearch for a year and never generated a certificate.

The demo certificates that ship with the package are for local testing only. Do not use them across two machines. Generate a small CA and one certificate per node, with the node's hostname and IP in the subject alternative names.

Then list the node certificate DNs on every node:

plugins.security.ssl.transport.pemcert_filepath: certs/search-01.pem
plugins.security.ssl.transport.pemkey_filepath: certs/search-01-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true

plugins.security.nodes_dn:
  - 'CN=search-01,OU=search,O=linkhub,L=Copenhagen,C=DK'
  - 'CN=search-02,OU=search,O=linkhub,L=Copenhagen,C=DK'

plugins.security.allow_unsafe_democertificates: false

Every node's DN goes in the list, on both nodes. A node whose DN is missing is treated as a client, not a cluster member, and will not join.

Two failure modes worth recognising in the logs. If you see certificate_unknown, the certificate is not trusted by the other node's CA. If you see No subject alternative names matching IP address, the node is connecting from an address the certificate does not cover. That second one is common on hosts with several interfaces or dual-stack networking, where the node uses an address you did not think to include.

If the existing node has security disabled

There is a setting for exactly this migration:

plugins.security_config.ssl_dual_mode_enabled: true

It lets a node accept both encrypted and unencrypted transport connections, so you can enable security across a cluster by rolling restart rather than all at once. It is meant to be temporary. Turn it off once both nodes are using the same configuration.

Turning on replicas

A second node does nothing for redundancy until your indexes have replicas. On a single-node cluster, replica shards cannot be allocated, so people often set number_of_replicas to 0 to get a green cluster and then forget.

PUT /_all/_settings
{ "index": { "number_of_replicas": 1 } }

And set it for future indexes in your template, or you will create the next index with no replica and not notice.

One replica is right for two nodes. Two replicas would need a third data node to place them on, and the extra copies would stay unassigned, leaving the cluster permanently yellow.

Checking that it worked

GET /_cat/nodes?v
GET /_cluster/health?pretty
GET /_cat/shards?v

What you want to see: two nodes listed, one marked with an asterisk as cluster manager, status green, and in the shard listing every shard appearing twice, once as p and once as r, on different nodes.

Yellow means primaries are allocated but some replicas are not. On a fresh two-node cluster that usually means replication has not finished yet, so wait and check again. If it stays yellow, look at why the replicas cannot be placed:

GET /_cluster/allocation/explain

That endpoint tells you exactly which rule is preventing allocation, which is far quicker than guessing.

The test that matters

Stop OpenSearch on node 2 and watch what happens on node 1.

You should see the cluster go yellow. Searches keep working, because node 1 holds a full copy of the data. Writes and state changes may fail, because the quorum is gone.

Then do it the other way round: stop node 1 and watch node 2. Same result.

That is the limitation from the top of this article, demonstrated on your own hardware. Whether it is acceptable depends on what the cluster does. For a search index you can rebuild from your primary database, an outage is an inconvenience. For anything where search being down means the site is down, find a third vote.

I have not run this exact configuration end to end yet. Test it on throwaway machines before you point it at data you care about, and check the setting names against the docs for your OpenSearch version, since some were renamed from the Elasticsearch originals.