Top Description Inners Fields Constructors Methods
org.apache.derby.impl.services.daemon

public Class IndexStatisticsDaemonImpl

extends Object
implements IndexStatisticsDaemon, Runnable
Class Inheritance
All Implemented Interfaces
java.lang.Runnable, org.apache.derby.iapi.services.daemon.IndexStatisticsDaemon
Imports
java.io.PrintWriter, java.sql.Connection, java.util.ArrayList, .List, org.apache.derby.catalog.UUID, org.apache.derby.catalog.types.StatisticsImpl, org.apache.derby.shared.common.error.StandardException, .ExceptionSeverity, .ShutdownException, org.apache.derby.iapi.db.Database, org.apache.derby.shared.common.reference.Property, .SQLState, org.apache.derby.iapi.services.context.ContextManager, .ContextService, org.apache.derby.iapi.services.daemon.IndexStatisticsDaemon, org.apache.derby.iapi.services.property.PropertyUtil, org.apache.derby.shared.common.sanity.SanityManager, org.apache.derby.shared.common.stream.HeaderPrintWriter, org.apache.derby.iapi.services.uuid.UUIDFactory, org.apache.derby.iapi.services.monitor.Monitor, org.apache.derby.iapi.sql.conn.LanguageConnectionContext, org.apache.derby.iapi.sql.depend.DependencyManager, org.apache.derby.iapi.sql.dictionary.ConglomerateDescriptor, .DataDictionary, .IndexRowGenerator, .StatisticsDescriptor, .TableDescriptor, org.apache.derby.iapi.sql.execute.ExecIndexRow, org.apache.derby.iapi.store.access.ConglomerateController, .GroupFetchScanController, .ScanController, .TransactionController, org.apache.derby.iapi.types.DataValueDescriptor, org.apache.derby.iapi.util.InterruptStatus

Daemon acting as a coordinator for creating and updating index cardinality statistics.

The need for updated statistics is currently determined when compiling a SELECT query. The unit of work is then scheduled with this daemon, and the work itself will be carried out in a separate thread. If the worker thread doesn't exist it is created, if it is idle the unit of work will be processed immediately, and if it is busy the unit of work has to wait in the queue.

The daemon code has a notion of a background task. If the update is run as a background task, it will try to affect other activity in the Derby database as little as possible. As far as possible, it will not set locks on the conglomerates it scans, and if it needs to take locks it will give up immediately if the locks cannot be obtained. In some cases it will also roll back to release locks already taken, ad then retry. Since we are accessing shared structures the background work may still interfere with the user activity in the database due to locking, but all such operations carried out by the daemon are of short duration.

The high level flow of an update to index statistics is:

  1. schedule update (the only action carried out by the user thread)
  2. for each index:
    1. scan index
    2. invalidate statements dependent on current statistics
    3. drop existing statistics
    4. add new statistics

List of possible improvements:

  1. Reduce potential impact of multiple invalidations (per table), probably by finding a way to invalidate only once after all indexes for a table have had their statistics updated. So far invalidation has proven to be the most difficult piece of the puzzle due to the interaction with the data dictionary and sensitivity to concurrent activity for the table.

Implementation notes: List of potential cleanups before going into a release:

  1. Consider removing all tracing code. May involve improving logging if parts of the trace output is valuable enough.

Nested and Inner Type Summary

Modifier and TypeClass and Description
private static class
IndexStatisticsDaemonImpl.KeyComparator

Support class used to compare keys when scanning indexes.

Field Summary

Modifier and TypeField and Description
private static final boolean
private static final boolean
private final ContextManager
ctxMgr

The context manager for the worker thread.

private boolean
daemonDisabled

Tells if the daemon has been disabled.

private LanguageConnectionContext
daemonLCC

The language connection context for the worker thread.

private final String
private final Database
db

The database object for the database we are handling automatic index statistics update for.

private final String
dbOwner

The name of the database owner.

private final boolean
doLog

Tells if logging is enabled.

private final boolean
doTrace

Tells if tracing is enabled.

private int
errorsConsecutive

Number of consecutive errors, used as a metric to decide if the damoen should be automatically shut down.

private long
private long
private final HeaderPrintWriter
private static final int
MAX_QUEUE_LENGTH

Maximum number of work units allowed in the queue.

private final ArrayList<TableDescriptor>
queue

A list of tables that shall have their index statistics updated.

private Thread
runningThread

The thread in which the index statistics refresh operation is being executed, if any.

private long
runTime

The period of time (ms) for which the daemon has been doing active work.

public final boolean
skipDisposableStats

Tells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior.

private final long
timeOfCreation

Specifies when the daemon was created.

private final boolean
traceToDerbyLog

Tells if traces are written to the Derby log file.

private final boolean
traceToStdOut

Tells if traces are written to standard out.

private final StringBuffer
private long
private long
private long
private long
private long

Constructor Summary

AccessConstructor and Description
public
IndexStatisticsDaemonImpl(HeaderPrintWriter
the log to write to
log
,
boolean
whether to log activity information
doLog
,
String
whether, and to where, trace information should be written ("off|log|stdout|both")
traceLevel
,
Database
the database ("off|log|stdout|both")
db
,
String
the name of the database owner
userName
,
String
the name of the database (not stored in the db obj)
databaseName
)

Creates a new daemon.

Method Summary

Modifier and TypeMethod and Description
private boolean

Returns:

true if work can be accepted, false if not.
acceptWork
(TableDescriptor
the table descriptor to check
td
)

Determines if the given work can be accepted.

private void
appendRunStats(StringBuffer
the string buffer to append to
sb
)

Appends runtime statistics to the given string buffer.

private static String

Returns:

A string.
cardToStr
(long[]
index cardinality
cardinality
)

Produces a textual representation of the cardinality numbers.

private boolean
dbAtLeast10_9(Database db)

Tells if the database is 10.9 or newer.

private static String
extractIstatInfo(Throwable t)

Purely for debugging, to avoid printing too much info.

private static String
fmtScanTimes(long[][] timings)

Format array of scan durations as a string.

private void
generateStatistics(LanguageConnectionContext
connection context to use to perform the work
lcc
,
TableDescriptor
target base table descriptor
td
)

Generates index statistics for all indexes associated with the given table descriptor.

private static ContextService
getContextService()

Must be private so that user code can't call this entry point.

private boolean

Returns:

true if the error was handled, false otherwise
handleExpectedErrors
(TableDescriptor td, StandardException
the exception to handle
se
)

Handles expected errors.

private boolean

Returns:

true if the error was handled, false otherwise
handleFatalErrors
(ContextManager
context manager
cm
,
StandardException
the exception to handle
se
)

Handles fatal errors that will cause the daemon to be shut down.

private boolean

Returns:

true if the error was handled, false otherwise
handleUnexpectedErrors
(TableDescriptor td, StandardException
the exception to handle
se
)

Handles unexpected errors.

private void
invalidateStatements(LanguageConnectionContext
connection context to use to perform the work
lcc
,
TableDescriptor
the table to invalidate for
td
,
boolean
whether the update is done automatically as part of a background task or if explicitly invoked by the user
asBackgroundTask
)

Performs an invalidation action for the given table (the event being statistics update).

private boolean
isShuttingDown()

Return true if we are being shutdown

private void
log(boolean asBackgroundTask, TableDescriptor td, String msg)

private void
log(boolean
true if logging for the background daemon automatically updating stats, false if not
asBackgroundTask
,
TableDescriptor
current table descriptor being worked on, may be null
td
,
Throwable
raised error, may be null
t
,
String
the message to log
msg
)

Logs the information given.

private void
logAlways(TableDescriptor
current table descriptor being worked on, may be null
td
,
Throwable
raised error, may be null
t
,
String
the message to log
msg
)

Logs the information given.

private void
processingLoop()

Main processing loop which will compute statistics until the queue of scheduled work units has been drained.

public void
run()

Implements java.lang.Runnable.run.

Drives the statistics generation.

public void
runExplicitly(LanguageConnectionContext
connection context to use to perform the work
lcc
,
TableDescriptor
the base table
td
,
ConglomerateDescriptor[]
the indexes to update (non-index conglomerates are ignored)
cds
,
String
the context in which the operation is run (i.e. 'ALTER TABLE', may be null)
runContext
)

Implements org.apache.derby.iapi.services.daemon.IndexStatisticsDaemon.runExplicitly.

Runs the statistics update sequence explicitly as requested by the user.

public void
schedule(TableDescriptor
base table descriptor to update index statistics for
td
)

Implements org.apache.derby.iapi.services.daemon.IndexStatisticsDaemon.schedule.

Schedules an update of the index statistics for the specified table.

private void
setHeapRowEstimate(TransactionController
transaction to use
tc
,
long
the heap table
tableId
,
long
estimate of number of rows in the table
rowEstimate
)

Sets the row estimate for the heap conglomerate.

private static void
sleep(long
target sleep time
ms
)

Puts the current thread to sleep for maximum ms milliseconds.

public void
private synchronized void
trace(int indentLevel, String msg)

private void
updateIndexStatsMinion(LanguageConnectionContext
language connection context used to perform the work
lcc
,
TableDescriptor
the table to update index stats for
td
,
ConglomerateDescriptor[]
the conglomerates to update statistics for (non-index conglomerates will be ignored), null means all indexes
cds
,
boolean
whether the updates are done automatically as part of a background task or if explicitly invoked by the user
asBackgroundTask
)

Updates the index statistics for the given table and the specified indexes.

private void
writeUpdatedStats(LanguageConnectionContext
connection context to use to perform the work
lcc
,
TableDescriptor
the base table
td
,
UUID
the index of the base table
index
,
long
number of rows in the base table
numRows
,
long[]
the number of unique values in the index (per number of leading columns)
cardinality
,
boolean
whether the update is done automatically as part of a background task or if explicitly invoked by the user
asBackgroundTask
)

Writes updated statistics for the specified index to the data dictionary.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

AS_BACKGROUND_TASKback to summary
private static final boolean AS_BACKGROUND_TASK
AS_EXPLICIT_TASKback to summary
private static final boolean AS_EXPLICIT_TASK
ctxMgrback to summary
private final ContextManager ctxMgr

The context manager for the worker thread.

daemonDisabledback to summary
private boolean daemonDisabled

Tells if the daemon has been disabled.

daemonLCCback to summary
private LanguageConnectionContext daemonLCC

The language connection context for the worker thread.

databaseNameback to summary
private final String databaseName
dbback to summary
private final Database db

The database object for the database we are handling automatic index statistics update for.

dbOwnerback to summary
private final String dbOwner

The name of the database owner.

doLogback to summary
private final boolean doLog

Tells if logging is enabled.

doTraceback to summary
private final boolean doTrace

Tells if tracing is enabled.

errorsConsecutiveback to summary
private int errorsConsecutive

Number of consecutive errors, used as a metric to decide if the damoen should be automatically shut down.

errorsKnownback to summary
private long errorsKnown
errorsUnknownback to summary
private long errorsUnknown
logStreamback to summary
private final HeaderPrintWriter logStream
MAX_QUEUE_LENGTHback to summary
private static final int MAX_QUEUE_LENGTH

Maximum number of work units allowed in the queue.

queueback to summary
private final ArrayList<TableDescriptor> queue

A list of tables that shall have their index statistics updated. Note that the descriptor isn't removed before the work has been completed.

runningThreadback to summary
private Thread runningThread

The thread in which the index statistics refresh operation is being executed, if any. Created as needed, but there will only be one thread doing the work. The thread is allowed to die since it is assumed that index statistics regeneration is rather infrequent.

runTimeback to summary
private long runTime

The period of time (ms) for which the daemon has been doing active work.

skipDisposableStatsback to summary
public final boolean skipDisposableStats

Tells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior.

Made public to allow access for CreateIndexConstantAction and FromBaseTable, but this is no longer necessary when the debug property to keep disposable statistics is removed.

timeOfCreationback to summary
private final long timeOfCreation

Specifies when the daemon was created.

traceToDerbyLogback to summary
private final boolean traceToDerbyLog

Tells if traces are written to the Derby log file.

traceToStdOutback to summary
private final boolean traceToStdOut

Tells if traces are written to standard out.

tsbback to summary
private final StringBuffer tsb
wuProcessedback to summary
private long wuProcessed
wuRejectedDupback to summary
private long wuRejectedDup
wuRejectedFQback to summary
private long wuRejectedFQ
wuRejectedOtherback to summary
private long wuRejectedOther
wuScheduledback to summary
private long wuScheduled

Constructor Detail

IndexStatisticsDaemonImplback to summary
public IndexStatisticsDaemonImpl(HeaderPrintWriter log, boolean doLog, String traceLevel, Database db, String userName, String databaseName)

Creates a new daemon.

Parameters
log:HeaderPrintWriter

the log to write to

doLog:boolean

whether to log activity information

traceLevel:String

whether, and to where, trace information should be written ("off|log|stdout|both")

db:Database

the database ("off|log|stdout|both")

userName:String

the name of the database owner

databaseName:String

the name of the database (not stored in the db obj)

Method Detail

acceptWorkback to summary
private boolean acceptWork(TableDescriptor td)

Determines if the given work can be accepted.

Parameters
td:TableDescriptor

the table descriptor to check

Returns:boolean

true if work can be accepted, false if not.

appendRunStatsback to summary
private void appendRunStats(StringBuffer sb)

Appends runtime statistics to the given string buffer.

Parameters
sb:StringBuffer

the string buffer to append to

cardToStrback to summary
private static String cardToStr(long[] cardinality)

Produces a textual representation of the cardinality numbers.

Parameters
cardinality:long[]

index cardinality

Returns:String

A string.

dbAtLeast10_9back to summary
private boolean dbAtLeast10_9(Database db)

Tells if the database is 10.9 or newer.

extractIstatInfoback to summary
private static String extractIstatInfo(Throwable t)

Purely for debugging, to avoid printing too much info.

fmtScanTimesback to summary
private static String fmtScanTimes(long[][] timings)

Format array of scan durations as a string.

generateStatisticsback to summary
private void generateStatistics(LanguageConnectionContext lcc, TableDescriptor td) throws StandardException

Generates index statistics for all indexes associated with the given table descriptor.

This method is run as a background task.

Parameters
lcc:LanguageConnectionContext

connection context to use to perform the work

td:TableDescriptor

target base table descriptor

Exceptions
StandardException:
if accessing the conglomerates fail
getContextServiceback to summary
private static ContextService getContextService()

Must be private so that user code can't call this entry point.

handleExpectedErrorsback to summary
private boolean handleExpectedErrors(TableDescriptor td, StandardException se)

Handles expected errors.

The logging of expected errors is for observability purposes only. The daemon is capable of dealing with these errors, and no interaction from the user is expected.

Parameters
se:StandardException

the exception to handle

Returns:boolean

true if the error was handled, false otherwise

handleFatalErrorsback to summary
private boolean handleFatalErrors(ContextManager cm, StandardException se)

Handles fatal errors that will cause the daemon to be shut down.

Parameters
cm:ContextManager

context manager

se:StandardException

the exception to handle

Returns:boolean

true if the error was handled, false otherwise

handleUnexpectedErrorsback to summary
private boolean handleUnexpectedErrors(TableDescriptor td, StandardException se)

Handles unexpected errors.

Unexpected errors are error conditions the daemon isn't set up to handle specifically. For this reason the stack trace will be logged to allow for later investigation.

In general it is expected that the daemon will be able to recover by dropping the current unit of work and move on to the next one (if any).

Parameters
se:StandardException

the exception to handle

Returns:boolean

true if the error was handled, false otherwise

invalidateStatementsback to summary
private void invalidateStatements(LanguageConnectionContext lcc, TableDescriptor td, boolean asBackgroundTask) throws StandardException

Performs an invalidation action for the given table (the event being statistics update).

Parameters
lcc:LanguageConnectionContext

connection context to use to perform the work

td:TableDescriptor

the table to invalidate for

asBackgroundTask:boolean

whether the update is done automatically as part of a background task or if explicitly invoked by the user

Exceptions
StandardException:
if the invalidation request fails
isShuttingDownback to summary
private boolean isShuttingDown()

Return true if we are being shutdown

logback to summary
private void log(boolean asBackgroundTask, TableDescriptor td, String msg)
See Also
log(boolean, TableDescriptor, Throwable, String)
logback to summary
private void log(boolean asBackgroundTask, TableDescriptor td, Throwable t, String msg)

Logs the information given.

Note that if asBackgroundTask is false, nothing will be logged currently.

Parameters
asBackgroundTask:boolean

true if logging for the background daemon automatically updating stats, false if not

td:TableDescriptor

current table descriptor being worked on, may be null

t:Throwable

raised error, may be null

msg:String

the message to log

logAlwaysback to summary
private void logAlways(TableDescriptor td, Throwable t, String msg)

Logs the information given.

Parameters
td:TableDescriptor

current table descriptor being worked on, may be null

t:Throwable

raised error, may be null

msg:String

the message to log

processingLoopback to summary
private void processingLoop()

Main processing loop which will compute statistics until the queue of scheduled work units has been drained.

runback to summary
public void run()

Implements java.lang.Runnable.run.

Drives the statistics generation.

This method will be run in a separate thread, and it will keep working as long as there is work to do. When the queue is exhausted, the method will exit (the thread dies).

runExplicitlyback to summary
public void runExplicitly(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, String runContext) throws StandardException

Implements org.apache.derby.iapi.services.daemon.IndexStatisticsDaemon.runExplicitly.

Runs the statistics update sequence explicitly as requested by the user.

Parameters
lcc:LanguageConnectionContext

connection context to use to perform the work

td:TableDescriptor

the base table

cds:ConglomerateDescriptor[]

the indexes to update (non-index conglomerates are ignored)

runContext:String

the context in which the operation is run (i.e. 'ALTER TABLE', may be null)

Exceptions
StandardException:
if updating the index statistics fails
scheduleback to summary
public void schedule(TableDescriptor td)

Implements org.apache.derby.iapi.services.daemon.IndexStatisticsDaemon.schedule.

Schedules an update of the index statistics for the specified table.

Assume the descriptor will be valid until we get around to generate the statistics. If it turns out to be invalid, it will be discarded.

Parameters
td:TableDescriptor

base table descriptor to update index statistics for

setHeapRowEstimateback to summary
private void setHeapRowEstimate(TransactionController tc, long tableId, long rowEstimate) throws StandardException

Sets the row estimate for the heap conglomerate.

Parameters
tc:TransactionController

transaction to use

tableId:long

the heap table

rowEstimate:long

estimate of number of rows in the table

Exceptions
StandardException:
if accessing the table fails
sleepback to summary
private static void sleep(long ms)

Puts the current thread to sleep for maximum ms milliseconds.

No guarantee is provided for the minimum amount of time slept. If interrupted, the interrupt flag will be set again.

Parameters
ms:long

target sleep time

stopback to summary
public void stop()

Implements org.apache.derby.iapi.services.daemon.IndexStatisticsDaemon.stop.

Stops the daemon.

Will also clear the queue and print runtime statistics to the log the first time the method is invoked.

traceback to summary
private synchronized void trace(int indentLevel, String msg)
updateIndexStatsMinionback to summary
private void updateIndexStatsMinion(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, boolean asBackgroundTask) throws StandardException

Updates the index statistics for the given table and the specified indexes.

API note: Using null to update the statistics for all conglomerates is preferred over explicitly passing an array with all the conglomerates for the table. Doing so allows for some optimizations, and will cause a disposable statistics check to be performed.

Parameters
lcc:LanguageConnectionContext

language connection context used to perform the work

td:TableDescriptor

the table to update index stats for

cds:ConglomerateDescriptor[]

the conglomerates to update statistics for (non-index conglomerates will be ignored), null means all indexes

asBackgroundTask:boolean

whether the updates are done automatically as part of a background task or if explicitly invoked by the user

Exceptions
StandardException:
if something goes wrong
writeUpdatedStatsback to summary
private void writeUpdatedStats(LanguageConnectionContext lcc, TableDescriptor td, UUID index, long numRows, long[] cardinality, boolean asBackgroundTask) throws StandardException

Writes updated statistics for the specified index to the data dictionary.

Parameters
lcc:LanguageConnectionContext

connection context to use to perform the work

td:TableDescriptor

the base table

index:UUID

the index of the base table

numRows:long

number of rows in the base table

cardinality:long[]

the number of unique values in the index (per number of leading columns)

asBackgroundTask:boolean

whether the update is done automatically as part of a background task or if explicitly invoked by the user

Exceptions
StandardException:
if updating the data dictionary fails
org.apache.derby.impl.services.daemon back to summary

private Class IndexStatisticsDaemonImpl.KeyComparator

extends Object
Class Inheritance
  • java.lang.Object
  • org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.KeyComparator

Support class used to compare keys when scanning indexes.

Field Summary

Modifier and TypeField and Description
private DataValueDescriptor[]
private static final int
FETCH_SIZE

Number of rows fetched per iteration.

private DataValueDescriptor[]
private long
private DataValueDescriptor[]
private final DataValueDescriptor[][]
private int

Constructor Summary

AccessConstructor and Description
public
KeyComparator(ExecIndexRow
index row (template)
ir
)

Creates a key comparator for the given index.

Method Summary

Modifier and TypeMethod and Description
public int

Returns:

-1 if the current and previous key are identical, the index of the changed part of the key otherwise ([0, key length>)
compareWithPrevKey
(int
row index
index
)

Compares the key at the specified index with the previous key.

public int

Returns:

Number of rows fetched.
fetchRows
(GroupFetchScanController
the scan controller
gsc
)

Fetches rows from the scan controller.

public long

Returns:

Number of rows fetched.
getRowCount
()

Returns the number of rows fetched.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

currback to summary
private DataValueDescriptor[] curr
FETCH_SIZEback to summary
private static final int FETCH_SIZE

Number of rows fetched per iteration.

lastUniqueKeyback to summary
private DataValueDescriptor[] lastUniqueKey
numRowsback to summary
private long numRows
prevback to summary
private DataValueDescriptor[] prev
rowBufferArrayback to summary
private final DataValueDescriptor[][] rowBufferArray
rowsReadLastReadback to summary
private int rowsReadLastRead

Constructor Detail

KeyComparatorback to summary
public KeyComparator(ExecIndexRow ir)

Creates a key comparator for the given index.

Parameters
ir:ExecIndexRow

index row (template)

Method Detail

compareWithPrevKeyback to summary
public int compareWithPrevKey(int index) throws StandardException

Compares the key at the specified index with the previous key.

Parameters
index:int

row index

Returns:int

-1 if the current and previous key are identical, the index of the changed part of the key otherwise ([0, key length>)

Exceptions
StandardException:
if comparing the two keys fails
fetchRowsback to summary
public int fetchRows(GroupFetchScanController gsc) throws StandardException

Fetches rows from the scan controller.

Parameters
gsc:GroupFetchScanController

the scan controller

Returns:int

Number of rows fetched.

Exceptions
StandardException:
if fetching rows fails
getRowCountback to summary
public long getRowCount()

Returns the number of rows fetched.

Returns:long

Number of rows fetched.