java.lang.Object
org.apache.lucene.index.DocumentsWriterDeleteQueue
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Accountable
DocumentsWriterDeleteQueue
is a non-blocking linked pending deletes queue. In contrast to
other queue implementation we only maintain the tail of the queue. A delete queue is always used
in a context of a set of DWPTs and a global delete pool. Each of the DWPT and the global pool
need to maintain their 'own' head of the queue (as a DeleteSlice instance per DocumentsWriterPerThread
). The difference between the DWPT and the global pool is that the DWPT
starts maintaining a head once it has added its first document since for its segments private
deletes only the deletes after that document are relevant. The global pool instead starts
maintaining the head once this instance is created by taking the sentinel instance as its initial
head.
Since each DocumentsWriterDeleteQueue.DeleteSlice
maintains its own head and the list is only single linked the
garbage collector takes care of pruning the list for us. All nodes in the list that are still
relevant should be either directly or indirectly referenced by one of the DWPT's private DocumentsWriterDeleteQueue.DeleteSlice
or by the global BufferedUpdates
slice.
Each DWPT as well as the global delete pool maintain their private DeleteSlice instance. In the DWPT case updating a slice is equivalent to atomically finishing the document. The slice update guarantees a "happens before" relationship to all other updates in the same indexing session. When a DWPT updates a document it:
- consumes a document and finishes its processing
- updates its private
DocumentsWriterDeleteQueue.DeleteSlice
either by callingupdateSlice(DeleteSlice)
oradd(Node, DeleteSlice)
(if the document has a delTerm) - applies all deletes in the slice to its private
BufferedUpdates
and resets it - increments its internal document id
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescription(package private) static class
private static final class
(package private) static class
private static final class
private static final class
private static final class
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate boolean
private boolean
(package private) final long
private final BufferedUpdates
(package private) final ReentrantLock
private final DocumentsWriterDeleteQueue.DeleteSlice
Used to record deletes against all prior (already written to disk) segments.private final InfoStream
private long
private final AtomicLong
Generates the sequence number that IW returns to callers changing the index, showing the effective serialization of all operations.private final LongSupplier
private final long
private DocumentsWriterDeleteQueue.Node<?>
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
Constructor Summary
ConstructorsModifierConstructorDescription(package private)
DocumentsWriterDeleteQueue
(InfoStream infoStream) private
DocumentsWriterDeleteQueue
(InfoStream infoStream, long generation, long startSeqNo, LongSupplier previousMaxSeqId) -
Method Summary
Modifier and TypeMethodDescription(package private) long
add
(DocumentsWriterDeleteQueue.Node<?> newNode) (package private) long
add
(DocumentsWriterDeleteQueue.Node<?> deleteNode, DocumentsWriterDeleteQueue.DeleteSlice slice) invariant for document update(package private) long
(package private) long
(package private) long
addDocValuesUpdates
(DocValuesUpdate... updates) (package private) DocumentsWriterDeleteQueue
advanceQueue
(int maxNumPendingOps) Advances the queue to the next queue on flush.(package private) boolean
(package private) void
clear()
void
close()
private void
(package private) FrozenBufferedUpdates
freezeGlobalBuffer
(DocumentsWriterDeleteQueue.DeleteSlice callerSlice) private FrozenBufferedUpdates
freezeGlobalBufferInternal
(DocumentsWriterDeleteQueue.Node<?> currentTail) int
(package private) long
(package private) long
Returns the maximum completed seq no for this queue.(package private) long
Returns the maximum sequence number for this queue.long
private static LongSupplier
getPrevMaxSeqIdSupplier
(AtomicLong nextSeqNo) (package private) boolean
Returnstrue
if it was advanced.boolean
isOpen()
(package private) FrozenBufferedUpdates
This may freeze the global buffer unless the delete queue has already been closed.(package private) static DocumentsWriterDeleteQueue.Node<DocValuesUpdate[]>
newNode
(DocValuesUpdate... updates) (package private) static DocumentsWriterDeleteQueue.Node<Term>
(package private) DocumentsWriterDeleteQueue.DeleteSlice
newSlice()
int
long
Return the memory usage of this object in bytes.(package private) void
skipSequenceNumbers
(long jump) Inserts a gap in the sequence numbers.toString()
(package private) void
(package private) long
Negative result means there were new deletes since we last applied(package private) boolean
Just like updateSlice, but does not assign a sequence numberMethods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
Field Details
-
tail
-
closed
private volatile boolean closed -
globalSlice
Used to record deletes against all prior (already written to disk) segments. Whenever any segment flushes, we bundle up this set of deletes and insert into the buffered updates stream before the newly flushed segment(s). -
globalBufferedUpdates
-
globalBufferLock
-
generation
final long generation -
nextSeqNo
Generates the sequence number that IW returns to callers changing the index, showing the effective serialization of all operations. -
infoStream
-
maxSeqNo
private volatile long maxSeqNo -
startSeqNo
private final long startSeqNo -
previousMaxSeqId
-
advanced
private boolean advanced
-
-
Constructor Details
-
DocumentsWriterDeleteQueue
DocumentsWriterDeleteQueue(InfoStream infoStream) -
DocumentsWriterDeleteQueue
private DocumentsWriterDeleteQueue(InfoStream infoStream, long generation, long startSeqNo, LongSupplier previousMaxSeqId)
-
-
Method Details
-
addDelete
-
addDelete
-
addDocValuesUpdates
-
newNode
-
newNode
-
add
long add(DocumentsWriterDeleteQueue.Node<?> deleteNode, DocumentsWriterDeleteQueue.DeleteSlice slice) invariant for document update -
add
-
anyChanges
boolean anyChanges() -
tryApplyGlobalSlice
void tryApplyGlobalSlice() -
freezeGlobalBuffer
-
maybeFreezeGlobalBuffer
FrozenBufferedUpdates maybeFreezeGlobalBuffer()This may freeze the global buffer unless the delete queue has already been closed. If the queue has been closed this method will returnnull
-
freezeGlobalBufferInternal
private FrozenBufferedUpdates freezeGlobalBufferInternal(DocumentsWriterDeleteQueue.Node<?> currentTail) -
newSlice
DocumentsWriterDeleteQueue.DeleteSlice newSlice() -
updateSlice
Negative result means there were new deletes since we last applied -
updateSliceNoSeqNo
Just like updateSlice, but does not assign a sequence number -
ensureOpen
private void ensureOpen() -
isOpen
public boolean isOpen() -
close
public void close()- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
-
numGlobalTermDeletes
public int numGlobalTermDeletes() -
clear
void clear() -
getBufferedUpdatesTermsSize
public int getBufferedUpdatesTermsSize() -
ramBytesUsed
public long ramBytesUsed()Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-
toString
-
getNextSequenceNumber
public long getNextSequenceNumber() -
getLastSequenceNumber
long getLastSequenceNumber() -
skipSequenceNumbers
void skipSequenceNumbers(long jump) Inserts a gap in the sequence numbers. This is used by IW during flush or commit to ensure any in-flight threads get sequence numbers inside the gap -
getMaxCompletedSeqNo
long getMaxCompletedSeqNo()Returns the maximum completed seq no for this queue. -
getPrevMaxSeqIdSupplier
-
advanceQueue
Advances the queue to the next queue on flush. This carries over the the generation to the next queue and set thegetMaxSeqNo()
based on the given maxNumPendingOps. This method can only be called once, subsequently the returned queue should be used.- Parameters:
maxNumPendingOps
- the max number of possible concurrent operations that will execute on this queue after it was advanced. This corresponds the the number of DWPTs that own the current queue at the moment when this queue is advanced since each these DWPTs can increment the seqId after we advanced it.- Returns:
- a new queue as a successor of this queue.
-
getMaxSeqNo
long getMaxSeqNo()Returns the maximum sequence number for this queue. This value will change once this queue is advanced. -
isAdvanced
boolean isAdvanced()Returnstrue
if it was advanced.
-