Conf

Good Day!!

Thank You for your patience and co-ordination.

From the below solr logs we suspect that Indexed data in Shard "CALSecurityLogs_shard8_replica2" might have got corrupted

{{
2017-10-22 14:07:04,427 ERROR org.apache.solr.servlet.SolrDispatchFilter: null:org.apache.solr.common.SolrException: SolrCore 'CALSecurityLogs_shard8_replica2' is not available due to init failure: Error opening new searcher
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:783)
        at org.apache.solr.servlet.SolrDispatchFilter.httpSolrCall(SolrDispatchFilter.java:362)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:255)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:408)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:622)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:291)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
        at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:413)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at
Caused by: java.io.EOFException: read past EOF: _b6eo.si
        at org.apache.solr.store.blockcache.CustomBufferedIndexInput.refill
}}

In looking at your solr configs, solr.hdfs.blockcache.write.enabled is set to true, and that would definitely contribute to the issues we are seeing.

That is noted in the known release issues for Search https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cloudera.com%2Fdocumentation%2Fenterprise%2Frelease-notes%2Ftopics%2Fcdh_rn_search_ki.html%23known_issues__section_yjg_mbk_ln&data=02%7C01%7Cyehudako%40matrixbi.co.il%7Cb98559ef5ce2481afa1108d51ba1e32d%7C8b3747c3cccd4b178f18bd610edc86e6%7C0%7C0%7C636445304488866387&sdata=tkTDJnz0A4rT0gl%2Bfux7L6lzvQa4ehKAJXZ9UlvtkPI%3D&reserved=0

It is recommended to turn off blockcache.write

To do so, you would need to do the following (where collection_name is the name of your:

1. get the current instancedir from ZK:
solrctl instancedir --get CALSecurityLogs /path/to/collection

2. Change the solr.hdfs.blockcache.write.enabled to false in solrconfig.xml:
<bool name="solr.hdfs.blockcache.write.enabled">${solr.hdfs.blockcache.write.enabled:false}</bool>

3. upload your instancedir back to ZK after changing the value from true to false:
solrctl instancedir --update CALSecurityLogs /path/to/collection

4. Reload the solr collection:
solrctl collection --reload CALSecurityLogs

5. Verify through the Solr UI that the value is changed correctly:
Go to https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fyour.solr.server%3A8983&data=02%7C01%7Cyehudako%40matrixbi.co.il%7Cb98559ef5ce2481afa1108d51ba1e32d%7C8b3747c3cccd4b178f18bd610edc86e6%7C0%7C0%7C636445304488866387&sdata=NWBYRgwPF2bAxm4ZjGbDmgdFWA5n6o0O7oETrBE8Rsg%3D&reserved=0, select your core, and then select config and verify that solr.hdfs.blockcache.write.enabled is defaulted to false.

6. Restart the solr instances (if you need to do a rolling restart, you can restart each one separately)

Once that is done, it would be helpful to monitor the logs to confirm there are no further issues.

=====================
Enabling blockcache writing may result in unusable indexes

It is possible to create indexes with solr.hdfs.blockcache.write.enabled set to true. Such indexes may appear corrupt to readers, and reading these indexes may irrecoverably corrupt indexes. Blockcache writing is disabled by default.

Workaround: Do not enable blockcache writing.

Bug ID: CDH-17978