My only intention to get your team unblocked, if the issue is preventing any Nifi concerns. Note that the above answer is provided from Lucene perspective as I am not a Nifi Expert. Wish to check if your Team have attempted to increase the OpenFileLimit of the User running the Nifi Process to manage the FileSystemException with "Too Many Open Files" & restart Nifi, which I assume would restart the Lucene Cores as well.
Once Alread圜losedException is reported, Restarting the Lucene Service would ensure the Cores are initialized afresh. It appears Nifi Provenance Repository uses Lucene for indexing & the Alread圜losedException means the Lucene Core being accessed has been Closed already, owing to FileSystemException with "Too Many Open Files" for the one of the Core Content "/provenance_repo/provenance_repository/lucene-8-index-1647749380623/_vd_Lucene80_0.dvd". Hello & we wait for our Nifi Guru Wish to provide a bit of information on the Lucene Exception. Alread圜losedException: this IndexWriter is closedĪt .IndexWriter.ensureOpen(IndexWriter.java:877)Īt .IndexWriter.ensureOpen(IndexWriter.java:891)Īt .IndexWriter.updateDocuments(IndexWriter.java:1468)Īt .IndexWriter.addDocuments(IndexWriter.java:1444)Īt .(LuceneEventIndexWriter.java:70)Īt .(EventIndexTask.java:202)Īt .(EventIndexTask.java:113)Īt $RunnableAdapter.call(Executors.java:511)Īt .run(FutureTask.java:266)Īt .runWorker(ThreadPoolExecutor.java:1149)Īt $n(ThreadPoolExecutor.java:624)Ĭaused by: : /provenance_repo/provenance_repository/lucene-8-index-1647749380623/_vd_Lucene80_0.dvd: Too many open filesĪt .Uni圎anslateToIOException(Uni圎xception.java:91)Īt .Uni圎xception.rethrowAsIOException(Uni圎xception.java:102)Īt .Uni圎xception.rethrowAsIOException(Uni圎xception.java:107)Īt .UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)Īt .FileSystemProvider.newOutputStream(FileSystemProvider.java:434)Īt .newOutputStream(Files.java:216)Īt .FSDirectory$FSIndexOutput.(FSDirectory.java:410)Īt .FSDirectory$FSIndexOutput.(FSDirectory.java:406)Īt .FSDirectory.createOutput(FSDirectory.java:254)Īt .LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)Īt .TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)Īt .lucene80.Lucene80DocValuesConsumer.(Lucene80DocValuesConsumer.java:79)Īt .(Lucene80DocValuesFormat.java:161)Īt .perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227)Īt .perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163)Īt .perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109)Īt .NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108)Īt .DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345)Īt .DefaultIndexingChain.flush(DefaultIndexingChain.java:225)Īt .DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)Īt .DocumentsWriter.doFlush(DocumentsWriter.java:476)Īt .DocumentsWriter.flushAllThreads(DocumentsWriter.java:656)Īt .IndexWriter.prepareCommitInternal(IndexWriter.java:3365)Īt .mitInternal(IndexWriter.java:3771)Īt .mit(IndexWriter.java:3729)Īt .(LuceneEventIndexWriter.java:101)Īt .(EventIndexTask.java:253)Īt .(EventIndexTask.java:232) 23:51:22,905 ERROR o.a.n.p. Failed to index Provenance Events
Nifi provenance data is not visible and remains at 80% in searching for provenance events after upgrading from 1.13.0 to 1.15.3