Troubleshooting Elasticsearch Configuration

This appendix describes some of the administrative functions you can perform on Elasticsearch using the command prompt and how to handle some common errors.

This section describes some of the administrative tasks you can perform in Elasticsearch using the command prompt.

Adding or Deleting a Node

When you add or delete a node, the shards and replicas are dynamically distributed across the available nodes.

Note: The number of replicas determine the number of nodes that can go down without any data loss. The maximum number of replicas should be less than the number of nodes, that is, one less than the number of nodes. Replicas that are equal to or greater than the number of nodes do not add any benefit to the cluster, rather indexing will fail in such a scenario.

Changing the Node Type

When you deploy the Elasticsearch search engine using the DPK that PeopleSoft delivers, by default the node type is set to master-data type.

If you want to change the node type, you need to update the elasticsearch.yml configuration file.

To specify a data node, for example, set

node.master: false
node.data: true

You should specify this setting before the node is brought up.

Viewing the List of Indexes in an Elasticsearch Cluster

After deploying search definitions and building indexes, you can find out the list of indexes present in your ES Cluster by executing the following command in the browser:

http(s)://host:port/_cat/indices?v

You can use a similar command to view the plug-ins in your Elasticsearch implementation:

https(s)://host:port?_cat/plugins?v

If you want to troubleshoot the PeopleSoft Search Framework and Elasticsearch integration or any search related issues, Oracle PeopleSoft recommends, as an initial step, to check the following:

  • Ensure that the integration of Elasticsearch with PeopleSoft Search Framework is correctly completed. You can check the integration using the Ping, Login Test, and Proxy Test and Validate buttons on the Search Instance Properties page in PIA.

  • If you’ve installed Kibana and configured it, you can use the Indexing Metrics dashboard to view whether the indexing process is successful or not. If the Transaction Status visualization displays errors in indexing, click the error slice of the pie-chart, and obtain the process instance ID and the index name.

  • Check the run control ID on the Schedule Search Index page for any errors.

    • In case of an error, the View Error link gives you a clue on the error.

    • The Schedule Search Index page also displays a link where you can view the IB transactions from asynchronous service monitoring.

    • If the Schedule Search Index page has no details, click the Message Log link and follow the steps in it.

  • Check the Integration Broker queues.

    Note: If Full Direct Transfer is enabled, then Integration Broker is bypassed. You should check the Indexing Metrics dashboard for indexing errors. If Full Direct Transfer is not enabled, the Indexing Metrics dashboard does not capture data, so you should check the Integration Broker queues for indexing of documents without attachments.

    • Navigate to the Asynchronous Services page (PeopleTools, Search Framework, Search Admin Activity Guide, Administration, Asynchronous Services) and in the Queue Level field, select Sub Contract.

    • Verify the Queue status for the PTSF_ES_SEND_Q queue.

    • If PTSF_ES_SEND_Q status is New, verify whether the IB Pub/Sub domain is active or not and activate if required (by selecting PeopleTools, Search Framework, Search Admin Activity Guide, Administration, Domain Status).

      Ensure that there are no previous error transaction with the same subqueue name.

    • If PTSF_ES_SEND_Q status is Error, read the error message.

      If the error was due to unavailability of the Elasticsearch server, that is, no connectivity with IB, you may resubmit the transaction.

      If you need more clarity on the error and need to troubleshoot further, activate the IB message log to level 5 and submit the transaction, and watch for the response from Elasticsearch.

  • Check the PeopleSoft MsgLog to identify reported errors.

    MsgLog is located in <PIA_HOME>\webserv\<DOMAIN>\applications\peoplesoft\PSIGW.war\WEB-INF folder.

  • Check the Elasticsearch logs to identify reported errors.

    Elasticsearch logs are logged by default in the ES_HOME/logs folder. However, you can specify the location of the log folder in the elasticsearch.yml configuration file.

Errors in MsgLog

Error

Possible Cause

Resolution

{"error":{"root_cause":[{"type":"orcl_auth_plugin_exception","reason":"Invalid user/password"}],"type":"orcl_auth_plugin_exception","reason":"Invalid user/password"},"status":401}

This might be due to invalid proxy user and password.

In PIA, navigate to the Search Instance Properties page and in the Search Instance Properties section, ensure that the proxy user and password are entered correctly. Click the Proxy Login button and if the credentials are correct, you should see a login success message.

":{"type":"unavailable_shards_exception","reason":"[orcl_es_defaultindex_h9285505][0] Not enough active copies to meet write consistency of [QUORUM] (have 1, needed 2). Timeout: [1m], request: [BulkShardRequest to [orcl_es_defaultindex_h9285505] containing [304] requests]"}}}

This message is displayed because the number of replicas are more than the number of nodes.

In PIA, navigate to the Search Options page and update the replicas to a number that is less than or equal to the number of nodes in the cluster.

Errors in Elasticsearch Logs

Error

Possible Cause

Resolution

ConnectTransportException[[host][ipaddress:9300] connect_timeout[30s]]; nested: ConnectTimeoutException[connection timed out: /ipaddress:9300]; at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:987)

This is due to one node trying to connect other node to form the cluster. However, the IP address of the other node is not reachable.

Navigate to $ES_HOME/config and ensure that the network.host and disscovery.zen.ping.unicast.hosts values contain the correct machine details.

org.elasticsearch.transport.BindTransportException: Failed to resolve host

Elasticsearch network host is set to a host name or IP address that is not resolvable.

Set it to the IP address of the machine where Elasticsearch is running.

RemoteTransportException-EsRejectedExecutionException

Set the threadpool.bulk.queue_size value to a value higher than the number of concurrent bulk requests you want to send.

Errors on PIA

Error

Possible Cause

Resolution

Integration Gateway - External System Contact Error

Integration Gateway was not able to contact the external system. The network location specified may be incorrect, or the site is permanently or temporarily down.

This error might be due to no connectivity to the Elasticsearch server.

In PIA, navigate to the Search Instance Properties page, and ensure that the correct host name and port are entered. Click the Ping button and verify whether the Elasticsearch server is up and running.

Search Instance Ping not working.

This error might be due to the following reasons:

  • Incorrect configuration of Integration Broker.

  • Exception in Elasticsearch.

  • Elasticsearch server is down or invalid network host.

  • Incorrect configuration of search instance.

  1. Check if Integration Broker is configured correctly and ping the IB node to verify. If IB is correctly configured, check whether the Elasticsearch server is up and running. You can ping the Elasticsearch server from any browser by entering the server and port, for example, http://server:port.

  2. If the ping from a browser is not successful, but the Elasticsearch server is up, ensure that the network.host property in the elasticsearch.yml configuration file found in ES_HOME/config/ contains the correct server name (if you host your own Elasticsearch server).

Deploy is throwing errors.

This error might be due to one or more search definitions in deployed state, but unavailable in Elasticsearch.

  1. Check if any other search definition is already deployed. If so, select the search definition and click Report Sync. If the definitions are unavailable in the server, reset the definitions so that you can deploy them again.

  2. If you do not want to do Step 1 and you need to deploy only a few definitions, then in the Index Settings, click the override option and change the Index name to the search definition name and deploy.

Index schedule status shows success, but search results are not displayed.

This message may be due to security of the search user or the security call back may not be responding with the correct values.

Resolution: 1. 2. 3. If the security response is empty or does not contain the security values applicable then revisit the security given to the user.

  1. Ensure that some documents are indexed in Elasticsearch. Use the Elasticsearch Interact page to check whether documents are indexed in Elasticsearch.

  2. If security is enabled on a search definition, then set the IB logging to 5 and then perform the test again. Verify whether a security call back took place.

  3. If the security response is empty or does not contain the required security values, then check the security applied for the user.

Unable to deploy a search definition due to validation_exception.

Sample error message:

Unable to deploy\ index search definition <search definition name> due to error Index creation Failed : validation_exception (262,2012)
PTSF_ES.ESIMPL.AdminService.OnExecute Name:AddIndexSettings  PCPC:25544  Statement:345
Called from:PTSF_ES.ESIMPL.AdminService.OnExecute Name:AddPSFTSource  Statement:902
Called from:PTSF_ES.ESIMPL.AdminService.OnExecute Name:AddPSFTSources  Statement:909
Called from:PTEM_CONFIG.PTSFAdministerSearch.OnExecute Name:deployPeoplesoftSource  Statement:519
Called from:PTEM_CONFIG.PTSFAdministerSearch.OnExecute Name:deploySearchDefiniton  Statem (0,0) PTSF_ES.ESIMPL.AdminService.OnExecute 
Name:AddIndexSettings  PCPC:25664  Statement:350

This error is due to the soft limit on the number of shards to 1000, which is introduced in Elasticsearch 7.0.

Resolution - increase the number of shards per node as required.

To increase the number of shards per node, for example, to 1500, use this command:

CURL –XPUT http://<eshost>:<port>/_cluster/settings

and in the body type the following line:

{"persistent":{"cluster.max_shards_per_node":1500}}