Memory limitation when using tMatchIndexPredict for Apache Spark Streaming
About this task
When the tMatchIndexPredict component is used
in a Spark Streaming Job, it exchanges data with the Elasticsearch server
repeatedly. The request and response data are stored in the buffer cache. When the
maximal memory size of the Job execution is reached, the following error
occurs:
Exception in thread "I/O dispatcher 1329" java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)