Data Migration Monitoring Metrics

If your DM cluster is deployed using TiUP, the monitoring system is also deployed at the same time. This document describes the monitoring metrics provided by DM-worker.

Task

In the Grafana dashboard, the default name of DM is DM-task.

overview

Overview contains some monitoring metrics of all the DM-worker and DM-master instances or sources in the currently selected task. The current default alert rule is only for a single DM-worker/DM-master instance/source.

Metric nameDescriptionAlertSeverity level
task stateThe state of subtasks for migrationN/AN/A
storage capacityThe total storage capacity of the disk occupied by relay logsN/AN/A
storage remainThe remaining storage capacity of the disk occupied by relay logsN/AN/A
binlog file gap between master and relayThe number of binlog files by which the relay processing unit is behind the upstream masterN/AN/A
load progressThe percentage of the completed loading process of the load unit. The value is between 0%~100%N/AN/A
binlog file gap between master and syncerThe number of binlog files by which the binlog replication unit is behind the upstream masterN/AN/A
shard lock resolvingWhether the current subtask is waiting for sharding DDL migration. A value greater than 0 means that the current subtask is waiting for sharding DDL migrationN/AN/A

Operation errors

Metric nameDescriptionAlertSeverity level
before any operate errorThe number of errors before any operationN/AN/A
source bound errorThe number of errors of data source binding operationsN/AN/A
start errorThe number of errors during the start of a subtaskN/AN/A
pause errorThe number of errors during the pause of a subtaskN/AN/A
resume errorThe number of errors during the resuming of a subtaskN/AN/A
auto-resume errorThe number of errors during the auto-resuming of a subtaskN/AN/A
update errorThe number of errors during the update of a subtaskN/AN/A
stop errorThe number of errors during the stop of a subtaskN/AN/A

High availability

Metric nameDescriptionAlertSeverity level
number of dm-masters start leader components per minuteThe number of DM-master attempts to enable leader related components per minuteN/AN/A
number of workers in different stateThe number of DM-workers in different statesSome DM-worker(s) has (have) been offline for more than one hourcritical
workers' stateThe state of the DM-workerN/AN/A
number of worker event errorThe number of different types of DM-worker errorsN/AN/A
shard ddl error per minuteThe number of different types of sharding DDL errors per minuteAny sharding DDL error occurscritical
number of pending shard ddlThe number of pending sharding DDL operationsAny pending sharding DDL operation has existed for more than one hourcritical

Task state

Metric nameDescriptionAlertSeverity level
task stateThe state of subtasksAn alert occurs when the subtask has been in the Paused state for more than 20 minutescritical

Dump/Load unit

The following metrics show only when task-mode is in the full or all mode.

Metric nameDescriptionAlertSeverity level
load progressThe percentage of the completed loading process of the load unit. The value range is 0%~100%N/AN/A
data file sizeThe total size of the data files (includes the INSERT INTO statement) in the full data imported by the load unitN/AN/A
dump process exits with errorThe dump unit encounters an error within the DM-worker and exitsImmediate alertscritical
load process exits with errorThe load unit encounters an error within the DM-worker and exitsImmediate alertscritical
table countThe total number of tables in the full data imported by the load unitN/AN/A
data file countThe total number of data files (includes the INSERT INTO statement) in the full data imported by the load unitN/AN/A
transaction execution latencyThe latency of executing a transaction by the load unit (in seconds)N/AN/A
statement execution latencyThe duration of executing a statement by the load unit (in seconds)N/AN/A
remaining timeThe remaining time of replicating data by the load unit (in seconds)N/AN/A

Binlog replication

The following metrics show only when task-mode is in the incremental or all mode.

Metric nameDescriptionAlertSeverity level
remaining time to syncThe predicted remaining time it takes for syncer to be completely migrated with the upstream master (in minutes)N/AN/A
replicate lag gaugeThe latency time it takes to replicate the binlog from upstream to downstream (in seconds)N/AN/A
replicate lag histogramThe histogram of replicating the binlog from upstream to downstream (in seconds). Note that due to different statistical mechanisms, the data might be inaccurateN/AN/A
process exist with errorThe binlog replication unit encounters an error within the DM-worker and exitsImmediate alertscritical
binlog file gap between master and syncerThe number of binlog files by which the syncer processing unit is behind the upstream masterAn alert occurs when the number of binlog files by which the syncer processing unit is behind the upstream master exceeds one (>1) and the condition lasts over 10 minutescritical
binlog file gap between relay and syncerThe number of binlog files by which syncer is behind relayAn alert occurs when the number of binlog files by which the syncer processing unit is behind the relay processing unit exceeds one (>1) and the condition lasts over 10 minutescritical
binlog event QPSThe number of binlog events received per unit of time (this number does not include the events that need to be skipped)N/AN/A
skipped binlog event QPSThe number of binlog events received per unit of time that need to be skippedN/AN/A
read binlog event durationThe duration that the binlog replication unit reads the binlog from the relay log or the upstream MySQL (in seconds)N/AN/A
transform binlog event durationThe duration that the binlog replication unit parses and transforms the binlog into SQL statements (in seconds)N/AN/A
dispatch binlog event durationThe duration that the binlog replication unit dispatches a binlog event (in seconds)N/AN/A
transaction execution latencyThe duration that the binlog replication unit executes the transaction to the downstream (in seconds)N/AN/A
binlog event sizeThe size of a binlog event that the binlog replication unit reads from the relay log or the upstream MySQLN/AN/A
DML queue remain lengthThe length of the remaining DML job queueN/AN/A
total sqls jobsThe number of newly added jobs per unit of timeN/AN/A
finished sqls jobsThe number of finished jobs per unit of timeN/AN/A
statement execution latencyThe duration that the binlog replication unit executes the statement to the downstream (in seconds)N/AN/A
add job durationThe duration tht the binlog replication unit adds a job to the queue (in seconds)N/AN/A
DML conflict detect durationThe duration that the binlog replication unit detects the conflict in DML (in seconds)N/AN/A
skipped event durationThe duration that the binlog replication unit skips a binlog event (in seconds)N/AN/A
unsynced tablesThe number of tables that have not received the shard DDL statement in the current subtaskN/AN/A
shard lock resolvingWhether the current subtask is waiting for the shard DDL lock to be resolved. A value greater than 0 indicates that it is waiting for the shard DDL lock to be resolvedN/AN/A
ideal QPSThe highest QPS that can be achieved when the running time of DM is 0N/AN/A
binlog event rowThe number of rows in a binlog eventN/AN/A
finished transaction totalThe number of finished transactions in totalN/AN/A
replication transaction batchThe number of sql rows in the transaction executed to the downstreamN/AN/A
flush checkpoints time intervalThe time interval for flushing the checkpoints (in seconds)N/AN/A

Relay log

Note

Currently, DM v2.0 does not support enabling the relay log feature.

Metric nameDescriptionAlertSeverity level
storage capacityThe storage capacity of the disk occupied by the relay logN/AN/A
storage remainThe remaining storage capacity of the disk occupied by the relay logAn alert is needed once the value is smaller than 10Gcritical
process exits with errorThe relay log encounters an error within the DM-worker and exitsImmediate alertscritical
relay log data corruptionThe number of corrupted relay log filesImmediate alertsemergency
fail to read binlog from masterThe number of errors encountered when the relay log reads the binlog from the upstream MySQLImmediate alertscritical
fail to write relay logThe number of errors encountered when the relay log writes the binlog to disksImmediate alertscritical
binlog file indexThe largest index number of relay log files. For example, "value = 1" indicates "relay-log.000001"N/AN/A
binlog file gap between master and relayThe number of binlog files in the relay log that are behind the upstream masterAn alert occurs when the number of binlog files by which the relay processing unit is behind the upstream master exceeds one (>1) and the condition lasts over 10 minutescritical
binlog posThe write offset of the latest relay log fileN/AN/A
read binlog event durationThe duration that the relay log reads binlog from the upstream MySQL (in seconds)N/AN/A
write relay log durationThe duration that the relay log writes binlog into the disks each time (in seconds)N/AN/A
binlog event sizeThe size of a single binlog event that the relay log writes into the disksN/AN/A

Instance

In the Grafana dashboard, the default name of an instance is DM-instance.

Relay log

Metric nameDescriptionAlertSeverity level
storage capacityThe total storage capacity of the disk occupied by the relay logN/AN/A
storage remainThe remaining storage capacity within the disk occupied by the relay logAn alert occurs once the value is smaller than 10Gcritical
process exits with errorThe relay log encounters an error in DM-worker and exitsImmediate alertscritical
relay log data corruptionThe number of corrupted relay logsImmediate alertsemergency
fail to read binlog from masterThe number of errors encountered when relay log reads the binlog from the upstream MySQLImmediate alertscritical
fail to write relay logThe number of errors encountered when the relay log writes the binlog to disksImmediate alertscritical
binlog file indexThe largest index number of relay log files. For example, "value = 1" indicates "relay-log.000001"N/AN/A
binlog file gap between master and relayThe number of binlog files by which the relay processing unit is behind the upstream masterAn alert occurs when the number of binlog files by which the relay processing unit is behind the upstream master exceeds one (>1) and the condition lasts over 10 minutescritical
binlog posThe write offset of the latest relay log fileN/AN/A
read binlog durationThe duration that the relay log reads the binlog from the upstream MySQL (in seconds)N/AN/A
write relay log durationThe duration that the relay log writes the binlog into the disk each time (in seconds)N/AN/A
binlog sizeThe size of a single binlog event that the relay log writes into the disksN/AN/A

Task

Metric nameDescriptionAlertSeverity level
task stateThe state of subtasks for migrationAn alert occurs when the subtask has been paused for more than 10 minutescritical
load progressThe percentage of the completed loading process of the load unit. The value range is 0%~100%N/AN/A
binlog file gap between master and syncerThe number of binlog files by which the binlog replication unit is behind the upstream masterN/AN/A
shard lock resolvingWhether the current subtask is waiting for sharding DDL migration. A value greater than 0 means that the current subtask is waiting for sharding DDL migrationN/AN/A
Download PDF
What’s on this page
Was this page helpful?