The need for updated statistics is currently determined when compiling a SELECT query. The unit of work is then scheduled with this daemon, and the work itself will be carried out in a separate thread. If the worker thread doesn't exist it is created, if it is idle the unit of work will be processed immediately, and if it is busy the unit of work has to wait in the queue.
The daemon code has a notion of a background task. If the update is run as a background task, it will try to affect other activity in the Derby database as little as possible. As far as possible, it will not set locks on the conglomerates it scans, and if it needs to take locks it will give up immediately if the locks cannot be obtained. In some cases it will also roll back to release locks already taken, ad then retry. Since we are accessing shared structures the background work may still interfere with the user activity in the database due to locking, but all such operations carried out by the daemon are of short duration.
The high level flow of an update to index statistics is:
List of possible improvements:
Implementation notes: List of potential cleanups before going into a release:
| Modifier and Type | Class and Description |
|---|---|
| private static class | IndexStatisticsDaemonImpl.
Support class used to compare keys when scanning indexes. |
| Modifier and Type | Field and Description |
|---|---|
| private static final boolean | |
| private static final boolean | |
| private final ContextManager | ctxMgr
The context manager for the worker thread. |
| private boolean | daemonDisabled
Tells if the daemon has been disabled. |
| private LanguageConnectionContext | daemonLCC
The language connection context for the worker thread. |
| private final String | |
| private final Database | db
The database object for the database we are handling automatic index statistics update for. |
| private final String | dbOwner
The name of the database owner. |
| private final boolean | doLog
Tells if logging is enabled. |
| private final boolean | doTrace
Tells if tracing is enabled. |
| private int | errorsConsecutive
Number of consecutive errors, used as a metric to decide if the damoen should be automatically shut down. |
| private long | |
| private long | |
| private final HeaderPrintWriter | |
| private static final int | MAX_QUEUE_LENGTH
Maximum number of work units allowed in the queue. |
| private final ArrayList | queue
A list of tables that shall have their index statistics updated. |
| private Thread | runningThread
The thread in which the index statistics refresh operation is being executed, if any. |
| private long | runTime
The period of time (ms) for which the daemon has been doing active work. |
| public final boolean | skipDisposableStats
Tells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior. |
| private final long | timeOfCreation
Specifies when the daemon was created. |
| private final boolean | traceToDerbyLog
Tells if traces are written to the Derby log file. |
| private final boolean | traceToStdOut
Tells if traces are written to standard out. |
| private final StringBuffer | |
| private long | |
| private long | |
| private long | |
| private long | |
| private long |
| Access | Constructor and Description |
|---|---|
| public | IndexStatisticsDaemonImpl(HeaderPrintWriter
the log to write to log, boolean whether to log activity information doLog, String whether, and to where, trace information should be
written ("off|log|stdout|both") traceLevel, Database the database ("off|log|stdout|both") db, String the name of the database owner userName, String the name of the database (not stored in the db obj) databaseName)Creates a new daemon. |
| Modifier and Type | Method and Description |
|---|---|
| private boolean | Returns: true if work can be accepted, false if not.the table descriptor to check td)Determines if the given work can be accepted. |
| private void | appendRunStats(StringBuffer
the string buffer to append to sb)Appends runtime statistics to the given string buffer. |
| private static String | Returns: A string.index cardinality cardinality)Produces a textual representation of the cardinality numbers. |
| private boolean | |
| private static String | |
| private static String | |
| private void | generateStatistics(LanguageConnectionContext
connection context to use to perform the work lcc, TableDescriptor target base table descriptor td)Generates index statistics for all indexes associated with the given table descriptor. |
| private static ContextService | |
| private boolean | Returns: true if the error was handled, false otherwisethe exception to handle se)Handles expected errors. |
| private boolean | Returns: true if the error was handled, false otherwisecontext manager cm, StandardException the exception to handle se)Handles fatal errors that will cause the daemon to be shut down. |
| private boolean | Returns: true if the error was handled, false otherwisethe exception to handle se)Handles unexpected errors. |
| private void | invalidateStatements(LanguageConnectionContext
connection context to use to perform the work lcc, TableDescriptor the table to invalidate for td, boolean whether the update is done automatically as
part of a background task or if explicitly invoked by the user asBackgroundTask)Performs an invalidation action for the given table (the event being statistics update). |
| private boolean | |
| private void | |
| private void | log(boolean
true if logging for the background
daemon automatically updating stats, false if notcurrent table descriptor being worked on, may be td, Throwable nullraised error, may be t, String nullthe message to log msg)Logs the information given. |
| private void | logAlways(TableDescriptor
current table descriptor being worked on, may be td, Throwable nullraised error, may be t, String nullthe message to log msg)Logs the information given. |
| private void | processingLoop()
Main processing loop which will compute statistics until the queue of scheduled work units has been drained. |
| public void | |
| public void | runExplicitly(LanguageConnectionContext
connection context to use to perform the work lcc, TableDescriptor the base table td, ConglomerateDescriptor[] the indexes to update (non-index conglomerates are ignored) cds, String the context in which the operation is run (i.e.
'ALTER TABLE', may be runContext)null)Implements org. Runs the statistics update sequence explicitly as requested by the user. |
| public void | schedule(TableDescriptor
base table descriptor to update index statistics for td)Implements org. Schedules an update of the index statistics for the specified table. |
| private void | setHeapRowEstimate(TransactionController
transaction to use tc, long the heap table tableId, long estimate of number of rows in the table rowEstimate)Sets the row estimate for the heap conglomerate. |
| private static void | |
| public void | stop()
Implements org. Stops the daemon. |
| private synchronized void | |
| private void | updateIndexStatsMinion(LanguageConnectionContext
language connection context used to perform the work lcc, TableDescriptor the table to update index stats for td, ConglomerateDescriptor[] the conglomerates to update statistics for (non-index
conglomerates will be ignored), cds, boolean null means all indexeswhether the updates are done automatically as
part of a background task or if explicitly invoked by the user asBackgroundTask)Updates the index statistics for the given table and the specified indexes. |
| private void | writeUpdatedStats(LanguageConnectionContext
connection context to use to perform the work lcc, TableDescriptor the base table td, UUID the index of the base table index, long number of rows in the base table numRows, long[] the number of unique values in the index (per number
of leading columns) cardinality, boolean whether the update is done automatically as
part of a background task or if explicitly invoked by the user asBackgroundTask)Writes updated statistics for the specified index to the data dictionary. |
| AS_BACKGROUND_TASK | back to summary |
|---|---|
| private static final boolean AS_BACKGROUND_TASK | |
| AS_EXPLICIT_TASK | back to summary |
|---|---|
| private static final boolean AS_EXPLICIT_TASK | |
| ctxMgr | back to summary |
|---|---|
| private final ContextManager ctxMgr The context manager for the worker thread. | |
| daemonDisabled | back to summary |
|---|---|
| private boolean daemonDisabled Tells if the daemon has been disabled. | |
| daemonLCC | back to summary |
|---|---|
| private LanguageConnectionContext daemonLCC The language connection context for the worker thread. | |
| databaseName | back to summary |
|---|---|
| private final String databaseName | |
| db | back to summary |
|---|---|
| private final Database db The database object for the database we are handling automatic index statistics update for. | |
| dbOwner | back to summary |
|---|---|
| private final String dbOwner The name of the database owner. | |
| doLog | back to summary |
|---|---|
| private final boolean doLog Tells if logging is enabled. | |
| doTrace | back to summary |
|---|---|
| private final boolean doTrace Tells if tracing is enabled. | |
| errorsConsecutive | back to summary |
|---|---|
| private int errorsConsecutive Number of consecutive errors, used as a metric to decide if the damoen should be automatically shut down. | |
| errorsKnown | back to summary |
|---|---|
| private long errorsKnown | |
| errorsUnknown | back to summary |
|---|---|
| private long errorsUnknown | |
| logStream | back to summary |
|---|---|
| private final HeaderPrintWriter logStream | |
| MAX_QUEUE_LENGTH | back to summary |
|---|---|
| private static final int MAX_QUEUE_LENGTH Maximum number of work units allowed in the queue. | |
| queue | back to summary |
|---|---|
| private final ArrayList<TableDescriptor> queue A list of tables that shall have their index statistics updated. Note that the descriptor isn't removed before the work has been completed. | |
| runningThread | back to summary |
|---|---|
| private Thread runningThread The thread in which the index statistics refresh operation is being executed, if any. Created as needed, but there will only be one thread doing the work. The thread is allowed to die since it is assumed that index statistics regeneration is rather infrequent. | |
| runTime | back to summary |
|---|---|
| private long runTime The period of time (ms) for which the daemon has been doing active work. | |
| skipDisposableStats | back to summary |
|---|---|
| public final boolean skipDisposableStats Tells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior. Made public to allow access for CreateIndexConstantAction and FromBaseTable, but this is no longer necessary when the debug property to keep disposable statistics is removed. | |
| timeOfCreation | back to summary |
|---|---|
| private final long timeOfCreation Specifies when the daemon was created. | |
| traceToDerbyLog | back to summary |
|---|---|
| private final boolean traceToDerbyLog Tells if traces are written to the Derby log file. | |
| traceToStdOut | back to summary |
|---|---|
| private final boolean traceToStdOut Tells if traces are written to standard out. | |
| tsb | back to summary |
|---|---|
| private final StringBuffer tsb | |
| wuProcessed | back to summary |
|---|---|
| private long wuProcessed | |
| wuRejectedDup | back to summary |
|---|---|
| private long wuRejectedDup | |
| wuRejectedFQ | back to summary |
|---|---|
| private long wuRejectedFQ | |
| wuRejectedOther | back to summary |
|---|---|
| private long wuRejectedOther | |
| wuScheduled | back to summary |
|---|---|
| private long wuScheduled | |
| IndexStatisticsDaemonImpl | back to summary |
|---|---|
| public IndexStatisticsDaemonImpl(HeaderPrintWriter log, boolean doLog, String traceLevel, Database db, String userName, String databaseName) Creates a new daemon.
| |
| acceptWork | back to summary |
|---|---|
| private boolean acceptWork(TableDescriptor td) Determines if the given work can be accepted.
| |
| appendRunStats | back to summary |
|---|---|
| private void appendRunStats(StringBuffer sb) Appends runtime statistics to the given string buffer.
| |
| cardToStr | back to summary |
|---|---|
| private static String cardToStr(long[] cardinality) Produces a textual representation of the cardinality numbers.
| |
| dbAtLeast10_9 | back to summary |
|---|---|
| private boolean dbAtLeast10_9(Database db) Tells if the database is 10.9 or newer. | |
| extractIstatInfo | back to summary |
|---|---|
| private static String extractIstatInfo(Throwable t) Purely for debugging, to avoid printing too much info. | |
| fmtScanTimes | back to summary |
|---|---|
| private static String fmtScanTimes(long[][] timings) Format array of scan durations as a string. | |
| generateStatistics | back to summary |
|---|---|
| private void generateStatistics(LanguageConnectionContext lcc, TableDescriptor td) throws StandardException Generates index statistics for all indexes associated with the given table descriptor. This method is run as a background task.
| |
| getContextService | back to summary |
|---|---|
| private static ContextService getContextService() Must be private so that user code can't call this entry point. | |
| handleExpectedErrors | back to summary |
|---|---|
| private boolean handleExpectedErrors(TableDescriptor td, StandardException se) Handles expected errors. The logging of expected errors is for observability purposes only. The daemon is capable of dealing with these errors, and no interaction from the user is expected.
| |
| handleFatalErrors | back to summary |
|---|---|
| private boolean handleFatalErrors(ContextManager cm, StandardException se) Handles fatal errors that will cause the daemon to be shut down.
| |
| handleUnexpectedErrors | back to summary |
|---|---|
| private boolean handleUnexpectedErrors(TableDescriptor td, StandardException se) Handles unexpected errors. Unexpected errors are error conditions the daemon isn't set up to handle specifically. For this reason the stack trace will be logged to allow for later investigation. In general it is expected that the daemon will be able to recover by dropping the current unit of work and move on to the next one (if any).
| |
| invalidateStatements | back to summary |
|---|---|
| private void invalidateStatements(LanguageConnectionContext lcc, TableDescriptor td, boolean asBackgroundTask) throws StandardException Performs an invalidation action for the given table (the event being statistics update).
| |
| isShuttingDown | back to summary |
|---|---|
| private boolean isShuttingDown() Return true if we are being shutdown | |
| log | back to summary |
|---|---|
| private void log(boolean asBackgroundTask, TableDescriptor td, String msg) | |
| log | back to summary |
|---|---|
| private void log(boolean asBackgroundTask, TableDescriptor td, Throwable t, String msg) Logs the information given.
Note that if
| |
| logAlways | back to summary |
|---|---|
| private void logAlways(TableDescriptor td, Throwable t, String msg) Logs the information given.
| |
| processingLoop | back to summary |
|---|---|
| private void processingLoop() Main processing loop which will compute statistics until the queue of scheduled work units has been drained. | |
| run | back to summary |
|---|---|
| public void run() Implements java. Drives the statistics generation. This method will be run in a separate thread, and it will keep working as long as there is work to do. When the queue is exhausted, the method will exit (the thread dies). | |
| runExplicitly | back to summary |
|---|---|
| public void runExplicitly(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, String runContext) throws StandardException Implements org. Runs the statistics update sequence explicitly as requested by the user.
| |
| schedule | back to summary |
|---|---|
| public void schedule(TableDescriptor td) Implements org. Schedules an update of the index statistics for the specified table. Assume the descriptor will be valid until we get around to generate the statistics. If it turns out to be invalid, it will be discarded.
| |
| setHeapRowEstimate | back to summary |
|---|---|
| private void setHeapRowEstimate(TransactionController tc, long tableId, long rowEstimate) throws StandardException Sets the row estimate for the heap conglomerate.
| |
| sleep | back to summary |
|---|---|
| private static void sleep(long ms) Puts the current thread to sleep for maximum No guarantee is provided for the minimum amount of time slept. If interrupted, the interrupt flag will be set again.
| |
| stop | back to summary |
|---|---|
| public void stop() Implements org. Stops the daemon. Will also clear the queue and print runtime statistics to the log the first time the method is invoked. | |
| trace | back to summary |
|---|---|
| private synchronized void trace(int indentLevel, String msg) | |
| updateIndexStatsMinion | back to summary |
|---|---|
| private void updateIndexStatsMinion(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, boolean asBackgroundTask) throws StandardException Updates the index statistics for the given table and the specified indexes.
API note: Using
| |
| writeUpdatedStats | back to summary |
|---|---|
| private void writeUpdatedStats(LanguageConnectionContext lcc, TableDescriptor td, UUID index, long numRows, long[] cardinality, boolean asBackgroundTask) throws StandardException Writes updated statistics for the specified index to the data dictionary.
| |
| Modifier and Type | Field and Description |
|---|---|
| private DataValueDescriptor[] | |
| private static final int | FETCH_SIZE
Number of rows fetched per iteration. |
| private DataValueDescriptor[] | |
| private long | |
| private DataValueDescriptor[] | |
| private final DataValueDescriptor[][] | |
| private int |
| Access | Constructor and Description |
|---|---|
| public |
| Modifier and Type | Method and Description |
|---|---|
| public int | Returns: -1 if the current and previous key are identical,
the index of the changed part of the key otherwise
([0, key length>)row index index)Compares the key at the specified index with the previous key. |
| public int | Returns: Number of rows fetched.the scan controller gsc)Fetches rows from the scan controller. |
| public long |
| curr | back to summary |
|---|---|
| private DataValueDescriptor[] curr | |
| FETCH_SIZE | back to summary |
|---|---|
| private static final int FETCH_SIZE Number of rows fetched per iteration. | |
| lastUniqueKey | back to summary |
|---|---|
| private DataValueDescriptor[] lastUniqueKey | |
| numRows | back to summary |
|---|---|
| private long numRows | |
| prev | back to summary |
|---|---|
| private DataValueDescriptor[] prev | |
| rowBufferArray | back to summary |
|---|---|
| private final DataValueDescriptor[][] rowBufferArray | |
| rowsReadLastRead | back to summary |
|---|---|
| private int rowsReadLastRead | |
| KeyComparator | back to summary |
|---|---|
| public KeyComparator(ExecIndexRow ir) Creates a key comparator for the given index.
| |
| compareWithPrevKey | back to summary |
|---|---|
| public int compareWithPrevKey(int index) throws StandardException Compares the key at the specified index with the previous key.
| |
| fetchRows | back to summary |
|---|---|
| public int fetchRows(GroupFetchScanController gsc) throws StandardException Fetches rows from the scan controller.
| |
| getRowCount | back to summary |
|---|---|
| public long getRowCount() Returns the number of rows fetched.
| |