| Modifier and Type | Field and Description |
|---|---|
| private long | |
| private boolean | |
| private boolean | |
| private boolean | |
| private long | |
| private long | |
| private long | |
| private final AtomicLong | currentMassiveDumpMessageLogDumpNumber
Whenever we produce a massive dump log message we will give it a unit identifier |
| private final AtomicLong | currentTinyMessageLogDumpNumber
Whenever we produce a tiny dump log message we will give it a unit identifier |
| private long | dateWhenLastConcurrencyManagerStateFullDumpWasPerformed
Whenever we decide to log a massive dump of the state of the concurrency manager we need to make this date move forward. |
| private final Object | |
| private static final long | |
| private static final long | |
| private static final boolean | |
| public static final int | |
| public static final int | |
| public static final long | |
| public static final long | |
| private static final boolean | |
| private static final long | |
| private static final long | |
| private static final long | |
| private static final boolean | |
| public static final boolean | |
| public static final boolean | |
| private long | |
| private long | |
| private long | |
| private int | |
| private int | |
| public static final ConcurrencyUtil | |
| private final AtomicLong | stackTraceIdAtomicLong
When we are explaining where read locks were acquired, the first time we see a new stack trace we create a stack trace id. Then for all other read locks we just say in the massive please go have a look at stack trace xxx. |
| private final ThreadLocal | threadLocalDateWhenCurrentThreadLastComplainedAboutBeingStuckInDeadLock
Thread local variable that allows the current thread to know when was the last time that this specific thread produced the "tiny dump" indicating that the thread is stuck. |
| private boolean | |
| private boolean |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| public DeferredLockManager | Returns: a cloned deferred lock manager. The newly created deferred lock manager will have its vectors of cache keys holding references the same cache keys as the original object. The cache keys themselves are not cloned. That measn that the DeferredLockManager will be immuatable in terms of its vectors and held references. But the objects it refers to (e.g. cache keys) can be mutating all the time if new readers or active threads arrive.an object that is originating from the map of thread to deferred locks from the concurrency manager
class. We do not want to be carrying around the original object while try to make a dump/snapshot of
the current state of the concurrency manager since these objects are always mutating. Locks are being
acquired and released etc... All the tiem. The only objest thta will be stable are those of threads
involved ina dead lock. And those are the threads that matter the most to us anyway. deferredLockManagerOriginal)Clone an original |
| public Map | Returns: A clone of that map that is a relatively stable data structure to work with since no new threads will register in or out in the map nor will the DeferredLockManager values be changing. As for the read lock manager we have no assurance as to what is happening with the cache keys themselves refered by theDeferredLockManager values, the cache keys are always changing their metadata as new threads come
in to do work or finish doing work. So it is not a perfect snapshot of the state of the system, but it is
as close as we can get.the original map taken from the conrruency manager itself deferredLockManagersOriginal)The exact same thing as the |
| public static Map | Returns: a cloned mapthe original map we want to clone mapThreadToWaitOnAcquireOriginal)Clone the map of the method names that tells us justification where threads acquire locks. |
| public static Map | Returns: a cloned map of thread to object id primary keys that a thread committing might have changed.map of thread to the primary keys of of objects changed by a transaction in the commit phase. This is
the original map grabbed from the WriteLockManager. mapThreadToObjectIdWithWriteLockManagerChangesOriginal)Clone the information about threads that are in the write lock manager trying to commit and the object ids they are holding with some arbitrary changes. |
| public static Map | Returns: a cloned mapthe original map we want to clone mapThreadToWaitOnAcquireOriginal)Clone the static map of the concurrency manager that tells us about threads waiting to acquire locks. |
| public static Map | Returns: a cloned mapthe original map we want to clone mapThreadToWaitOnAcquireInsideWriteLockManagerOriginal)Clone the static map of the concurrency manager that tells us about threads waiting to acquire locks. |
| public Map | Returns: A clone of the readLockManagersOriginal. Essentially the map instance returned is new and independent and the valuesReadLockManager are also clones and independent. The only thing that is
not cloned here - whose state could be changing - are the cache key themselves. The cache keys pointed by
the vector ReadLockManager#getReadLocks() are the original values. So our clone
from the read lock manager is not a perfectly stable clone. It will not be blowing up telling us
concurrent access modification when we loop through the vector. But no one can guarnate the number of
readers on the cache key stays the same nor that the active thread on a cache key stays the same... Those
values can definitely be fluctuating (not ideal ... but it would be quite hard to get better than this).This the original map of read locks referred by the concurrency manager. This is a very bad platform
to work with because if for whatever reason not all threads are frozen and some are actualy managing
to complete their transactions the contents of this map are systematically changing with threds being
added in and removed. readLockManagersOriginal)To facilitate algorithms that want to dump a snapshot of the current state of the concurrency manager or to start a hunt for dead locks this api faciliates the boostraping logic of such algorithms by giving the algorithm a stable clone of the map of read locks that we know will not change throughout the time the algorithm is running. |
| public static Set | Returns: A cloned has set of threads waiting for their deferred locks to be resolved.the original set of threads that are waiting for deferred locks to be resolved. setThreadWaitingToReleaseDeferredLocksOriginal)A set of threads that are at the end of object building and are waiting for the deferred locks to be resolved. |
| public ConcurrencyManagerState | createConcurrencyManagerState(Map<Thread, DeferredLockManager> deferredLockManagers, Map<Thread, ReadLockManager> readLockManagersOriginal, Map<Thread, ConcurrencyManager> mapThreadToWaitOnAcquireOriginal, Map<Thread, String> mapThreadToWaitOnAcquireMethodNameOriginal, Map<Thread, Set<ConcurrencyManager>> mapThreadToWaitOnAcquireInsideWriteLockManagerOriginal, Map<Thread, ConcurrencyManager> mapThreadToWaitOnAcquireReadLockOriginal, Map<Thread, String> mapThreadToWaitOnAcquireReadLockMethodNameOriginal, Set<Thread> setThreadWaitingToReleaseDeferredLocksOriginal, Map<Thread, String> mapThreadsThatAreCurrentlyWaitingToReleaseDeferredLocksJustificationClone, Map<Thread, Set<Object>> mapThreadToObjectIdWithWriteLockManagerChangesOriginal)
create a DTO that tries to represent the current snapshot of the concurrency manager and write lock manager cache state |
| protected String | createInformationAboutAllResourcesAcquiredAndDeferredByAllThreads(ConcurrencyManagerState
and object that represents a snapshot of the current state of the concurrency manager. concurrencyManagerState)Log information about all threads tracked in the concurrency manager. |
| protected String | Returns: a string describing the thread provided. We can see the active locks, deferred locks, read locks etc... as well sa if the thread is waiting to acquire a specific cache key or waiting for build object to complete.the read lock manager for the current thread readLockManager, DeferredLockManager the lock manager for the current thread lockManager, Set<ConcurrencyManager> null if the current thread is not waiting to acquire a cache key otherwise the cachekey that the
current thread wants to acquire and that is making it block. This field evolved to be a set and not
just one cache key because when we needed to tweak the write lock manager code to report about why the
write lock manager is stuck we need it to create the map
waitingOnAcquireCacheKeys, ConcurrencyManager org.
whereby during a commit where entiteis are merged into the shared cache a thread might be trying to
grab several write locks. so here we have a mix between the concurrency manager cache key a thread
currently wants together with cache keys the write lock managed is not managing to grab.cache key the thread is failing to acquire in the
waitingOnAcquireReadCacheKey, boolean org.true if the curren thread is now blocked waiting to confirm the locks it deferred have finished
building the corresponding objects. threadWaitingToReleaseDeferredLocks, Thread the thread eing described thread, int just loop incremented index to help the dump log messages give the feeling of the current thread being
described and how many more threads are still to be described currentThreadNumber, int the total number of threads being described in a for loop totalNumberOfThreads, Set<Object> Null for all threads excep those that are currently about to commit and merge changes to the shared
cache. In this case it holds the primary keys of the objects that were changed by the transaction. The
write lock manager has been tweaked to store information about objects ids that the current thread has
in its hands and that will required for write locks to be acquired by a committing thread. This
information is especially interesting if any thread participating in a dead lock is getting stuck in
the acquisition of write locks as part of the commit process. This information might end up revealing
a thread that has done too many changes and is creating a bigger risk fo dead lock. The more resources
an individual thread tries to grab the worse it is for the concurrency layer. The size of the change
set can be interesting. writeManagerThreadPrimaryKeysWithChangesToBeMerged, String when a thread is stuck for more than 500 ms in the release defferred locks algorithm, the concurrency
manager starts try to justify why the method isBuildObjectComplete keeps returning false. This
information is important whenever the param thread waiting to release deferred locks is true waitingToReleaseDeferredLocksJustification)Build a string that tries to describe in as much detail as possible the resources associated to the current thread. |
| private String | createInformationAboutAllThreadsWaitingToAcquireCacheKeys(Map<Thread, Set<ConcurrencyManager>>
this a cloned map that has an association between thread and cache keys the thread would like to
acquire but cannot because there are readers on the cache key. The thread might be stuck either on the
concurrency manager or on the write lock manager. unifiedMapOfThreadsStuckTryingToAcquireWriteLock, Map<Thread, String> the name of the method that updated the
mapThreadToWaitOnAcquireMethodNameClone)org. If we
do not know the method name that created the trace then it must have been the
org.
. This is not obvious but essentially we trace the acquisition of write locks in to places. The first
is the map already mentioned in the concurrency manager. The second is the map
org.
for the purose of the massive dump we act as if there was a single unified map. However when the
MAP_THREAD_TO_WAIT_ON_ACQUIRE we not only add to this map the cache key we cannot acquire but also the
method name. When we work with the map the THREADS_TO_FAIL_TO_ACQUIRE_CACHE_KEYS
we just keep trace of the cache key that could not be acquired. This
THREADS_TO_FAIL_TO_ACQUIRE_CACHE_KEYS is currently only used in one spot so we
can avoid the trouble of adding even more tracing for this.In this page of log dumping information we want to give a summary to the user of threads that appear to be stuck doing an acquire of the cache key. |
| protected String | createInformationAboutAllThreadsWaitingToAcquireReadCacheKeys(Map<Thread, ConcurrencyManager>
this a cloned map that has an association between thread and cache keys the thread would like to
acquire for READING but cannot because there is some active thread (other than themselves) holding the cache key (e.g. for writing) mapThreadToWaitOnAcquireReadLockClone, Map<Thread, String> mapThreadToWaitOnAcquireReadLockMethodNameClone)In this page of log dumping information we want to give a summary to the user of threads that appear to be stuck doing an acquire of the cache key. |
| protected String | createInformationAboutAllThreadsWaitingToReleaseDeferredLocks(Set<Thread>
threads waiting for the release deferred lock process to complete. setThreadWaitingToReleaseDeferredLocksClone)Log information about threads not moving forward because they are waiting for the
|
| private String | createInformationAboutCacheKeysAndThreadsMakingUseOfTheCacheKey(Map<ConcurrencyManager, CacheKeyToThreadRelationships>
a map that we have constructed where the map keys are cache keys that are of some sort of interest to
one or more threads (e.g. cache keys with a read lock, acquired or deferred) mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey)Log information focusing on the different cache keys where threads have hooks on the thread. |
| private String | |
| public ReadLockAcquisitionMetadata | Returns: object that have all the context information to allow us to know when and where exactly this key acquisition took place.the cache key we are about to increment and acquire for reading concurrencyManager)The concurrency managers about to acquire a cache key. |
| private String | createStringWithSummaryOfActiveLocksOnThread(DeferredLockManager lockManager, String threadName)
Create a print of the ACTIVE locks associated to the DeferredLockManager. |
| private String | createStringWithSummaryOfDeferredLocksOnThread(DeferredLockManager
the deferred lock manager of the current thread lockManager, String threadName)The |
| private String | Returns: A big string summarizing all of the read locks the thread.this is hacky class we created to close a gap in eclipselink code whereby read access on cache keys is
not tracked. The only thing that happens is incrementing the nuber of readers but that is not
sufficient if we need to abort all read locks. readLockManager, String the thread for which we are logging the read locks acquired threadName)Relates to issue. |
| public String | Returns: A to string of the cache key (e.g. that we are trying to lock
|
| protected String | Returns: Return the string with the tiny message we logged on the server log. This message can be interesting if we decide to fire up an interrupted exceptionhow many ms have passed since the thread stopped moving elapsedTime, ConcurrencyManager the current cache key that the thread is trying to acquire or the object where the thread is waiting
for the release deferred locks . concurrencyManager, DeferredLockManager the lock manager lockManager, ReadLockManager the read lock manager readLockManager)We have a thread that is not evolving for quite some while. |
| public void | determineIfReleaseDeferredLockAppearsToBeDeadLocked(ConcurrencyManager concurrencyManager, final long
the start date of the while tru loop for releasing a deferred lock whileStartTimeMillis, DeferredLockManager lockManager, ReadLockManager readLockManager, boolean this flag is to allow the write lock manager to say that it is afraid of a concurrency exception being
fire up because the thread in a dead lock might be trying to do a commit and blowing these threads up
is most likely too dangerous and possibly the eclipselink code is not robust enough to code with such
scenarios We do not care so much about blowing up exception during object building but during
committing of transactions we are very afraid callerIsWillingToAllowInterruptedExceptionToBeFiredUpIfNecessary)Throw an interrupted exception if appears that eclipse link code is taking too long to release a deferred lock. |
| public void | dumpConcurrencyManagerInformationIfAppropriate()
Invoke the |
| protected void | dumpConcurrencyManagerInformationStep01(Map<Thread, DeferredLockManager>
static map coming from the concurrency manager telling us all the threds and their defferred locks and
active locks deferredLockManagers, Map<Thread, ReadLockManager> static map coming from the concurrency manager telling us all the threads and their read locks readLockManagersOriginal, Map<Thread, ConcurrencyManager> static map of threads that have registered themselves as waiting for some cache key mapThreadToWaitOnAcquireOriginal, Map<Thread, String> mapThreadToWaitOnAcquireMethodNameOriginal, Map<Thread, Set<ConcurrencyManager>> this map relates to the fact that the write lock manager during transaction commits is very illusive.
The write lock manger is not allowing itself to get stuck on acquiring any cache key. It uses waits
with timings and therefore the locks needed to write and that cannot be obtained are not appearing
inside our tracebility maps of the concurrency manager. We needed add the
mapThreadToWaitOnAcquireInsideWriteLockManagerOriginal, Map<Thread, ConcurrencyManager> mapThreadToWaitOnAcquireReadLockOriginal, Map<Thread, String> mapThreadToWaitOnAcquireReadLockMethodNameOriginal, Set<Thread> org.
but semantically this map is 100 percent the same thing as the mapThreadToWaitOnAcquireOriginal. It
still represents a thread wanting to grab a write lock and not managing to get it. Being stuck in that
step. Wo we will want to fuse together the (mapThreadToWaitOnAcquireOriginal and the
mapThreadToWaitOnAcquireInsideWriteLockManagerOriginal) to make our lives much easier.static map of threads that have stopped going deeped in the recursion of object building and are
waiting for the confirmation that some of the objects they needed to build are finished building. setThreadWaitingToReleaseDeferredLocksOriginal, Map<Thread, String> mapThreadsThatAreCurrentlyWaitingToReleaseDeferredLocksJustificationClone, Map<Thread, Set<Object>> The write lock manager has been tweaked to store information about objects ids that the current thread
has in its hands and that will required for write locks to be acquired by a committing thread. This
information is especially interesting if any thread participating in a dead lock is getting stuck in
the acquisition of write locks as part of the commit process. This information might end up revealing
a thread that has done too many changes and is creating a bigger risk fo dead lock. The more resources
an individual thread tries to grab the worse it is for the concurrency layer. The size of the change
set can be interesting. mapThreadToObjectIdWithWriteLockManagerChangesOriginal)The current working thread is having problems. |
| protected void | dumpConcurrencyManagerInformationStep02(ConcurrencyManagerState
a snapshot of the current state of the concurrency manager and threads accessing locks. concurrencyManagerState)Dump the server log all of the information that we managed to aggregate about the current state of the concurrency manager. |
| protected String | |
| private String | enrichGenerateThreadDump()
We simply copy pasted this code from the net to have some helper tool to generate thread dumps programatically when the event takes place. |
| public String | Returns: get the stack trace of the current thread.This helper API is created due to the problem of the corruption of the eclipselink cache. |
| private void | enrichGenerateThreadDumpForThreadInfo(StringWriter writer, ThreadInfo threadInfo)
Enrich a given string building with the the thread writer for a given thread info object. |
| public void | enrichMapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKeyInfoAboutActiveAndDeferredLocks(Map<ConcurrencyManager, CacheKeyToThreadRelationships>
the map we want to enrich with more information mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey, Map<Thread, DeferredLockManager> the cloned map with information about threads and their deferred locks. deferredLockManagerMapClone)Enrich our map map of cache key to threads having a relationship with that object in regards to active locks on the cache key and deferred locks on the cache key |
| public void | enrichMapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKeyInfoAboutReadLocks(Map<ConcurrencyManager, CacheKeyToThreadRelationships>
a map whose metadata we need to enrich mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey, Map<Thread, ReadLockManager> map cloned from the original map and that gives us a snapshot of threads that acquired read locks readLockManagerMapClone)Enrich the mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey based on the read locks |
| public void | enrichMapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKeyInfoThreadsStuckOnAcquire(Map<ConcurrencyManager, CacheKeyToThreadRelationships>
the map we are trying to enirhc with additional information mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey, Map<Thread, Set<ConcurrencyManager>> a map telling us about threads that at a certain point in time were not progressing anywhere because
they were waiting to acquire a write lock. These are threads either stuck on the concurrency manager
or in the write lock manager during a transaction commmit unifiedMapOfThreadsStuckTryingToAcquireWriteLock)Enrich the mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey by setting on the cache keys the threads that are stuck trying to acquire the cache key. |
| public void | enrichMapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKeyInfoThreadsStuckOnAcquireLockForReading(Map<ConcurrencyManager, CacheKeyToThreadRelationships>
the map we are trying to enirhc with additional information mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey, Map<Thread, ConcurrencyManager> a map telling us about threads that at a certain point in time were not progressing anywhere because
they were waiting to acquire a lock. mapThreadToWaitOnAcquireReadLockClone)Enrich the mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey by setting on the cache keys the threads that are stuck trying to acquire the cache key with a read lock. |
| public static void | enrichMapThreadToWaitOnAcquireInsideWriteLockManagerClone(Map<Thread, Set<ConcurrencyManager>>
this is the map we want o enrich mapThreadToWaitOnAcquireInsideWriteLockManagerClone, Map<Thread, ConcurrencyManager> this is the map whose entries we want to copy into the map to enrich mapThreadToWaitOnAcquireClone)We have two maps we are using to trace threads that are stuck acquiring locks. |
| protected CacheKeyToThreadRelationships | Returns: never returls null. If the cache key is not yet in the map a ney entry is returned.the cache key we are search for cacheKey, Map<ConcurrencyManager, CacheKeyToThreadRelationships> the map of cache key to concurrency manager locking metadata mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey)Helper method to make sure we never get null dto from the mapOfCacheKeyToDtosExplainingThreadExpectationsOnCacheKey. |
| public long | |
| private boolean | |
| public long | Returns: "eclipselink.concurrency.manager.build.object.complete.waittime" persistence property value.
|
| public long | |
| public long | |
| private int | |
| private long | |
| public long | getMaxAllowedFrequencyToProduceMassiveDumpLogMessage()
If the system is perceived to be frozen and not evolving any longer, we will allow that every so often (e.g. once a minute) the logic complaining that the thread is stuck and going nowhere logs a very big dump message where the FULL concurrency manager state is explained. |
| public long | Returns: the frequency with which we are allowed to create a tiny dump log messageJust like we have a massive dump log message see |
| public long | Returns: property to control how long we are willing to wait before firing up an exception
|
| public int | |
| public int | |
| public boolean | Returns: true if we are supposed to be firing up exception to abort the thread in a dead lock, false we are afraid of trying to abort the transaction and not managing to resolve the dead lock and prefer to system frozen and be forced into restarting it.
|
| public boolean | |
| public boolean | |
| public boolean | |
| public boolean | |
| public String | readLockManagerProblem01CreateLogErrorMessageToIndicateThatCurrentThreadHasNullReadLockManagerWhileDecrementingNumberOfReaders(final int
the current count of readers on the cache key about to be decremented currentNumberOfReaders, final int the number of readers of the cache key if we subtract one reader decrementedNumberOfReaders, ConcurrencyManager the cache key that is about to suffer a decrement on the number of readers cacheKey)If when we are decrement the counter of number of readers on a cache key we find ourselves lacking the read lock manager at the time of the decrement we want to log a big fat error on the server log protecting that the current thread is misbehaving. |
| public String | readLockManagerProblem02ReadLockManageHasNoEntriesForThread(ConcurrencyManager cacheKey, long threadId)
|
| public String | readLockManagerProblem03ReadLockManageHasNoEntriesForThread(ConcurrencyManager cacheKey, long threadId)
|
| public void | |
| public void | |
| public void | |
| public void | setAllowTakingStackTraceDuringReadLockAcquisition(boolean allowTakingStackTraceDuringReadLockAcquisition)
|
| public void | |
| public void | |
| public void | |
| public void | setMaxAllowedFrequencyToProduceMassiveDumpLogMessage(long maxAllowedFrequencyToProduceMassiveDumpLogMessage)
|
| public void | setMaxAllowedFrequencyToProduceTinyDumpLogMessage(long maxAllowedFrequencyToProduceTinyDumpLogMessage)
|
| public void | |
| public void | setNoOfThreadsAllowedToDoWriteLockManagerAcquireRequiredLocksInParallel(int noOfThreadsAllowedToDoWriteLockManagerAcquireRequiredLocksInParallel)
|
| public void | |
| public void | |
| public void | setUseSemaphoreToLimitConcurrencyOnWriteLockManagerAcquireRequiredLocks(boolean useSemaphoreToLimitConcurrencyOnWriteLockManagerAcquireRequiredLocks)
|
| public boolean |