* [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis
@ 2026-05-27 13:55 wang.yaxin
2026-05-27 13:57 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: wang.yaxin @ 2026-05-27 13:55 UTC (permalink / raw)
To: akpm, fan.yu9, yang.yang29; +Cc: corbet, linux-kernel, linux-doc, xu.xin16
From: Wang Yaxin <wang.yaxin@zte.com.cn>
Previously delaytop only showed average delays. This patch adds:
1. delay_max fields to track the maximum delay value for each delay type
(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
per task.
2. The -t/--type option displays only the specified delay type with avg/max
values side by side, allowing focused analysis:
delaytop -t cpu # Show only CPU delay with avg/max
delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
3. Wall-clock timestamp when each maximum delay occurred, displayed in the
MAX_TIMESTAMP column when using -t/--type option. This enables:
- Identifying the time when a process experienced an abnormal delay max
- Correlating delay max across multiple processes at the same timestamp
- Cross-referencing with logs, traces, or other metrics at that time
4. When using -t/--type option, tasks are sorted by maximum delay value in
descending order (largest delay first), enabling quick identification of
top N processes with highest delay spikes.
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
Wang Yaxin (3):
delaytop: add delay max for delaytop
delaytop: add timestamp of delay max
delaytop: sort by max delay to highlight top latency processes
Documentation/accounting/delay-accounting.rst | 43 ++++
tools/accounting/delaytop.c | 232 +++++++++++++++---
2 files changed, 247 insertions(+), 28 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 1/3] delaytop: add delay max for delaytop 2026-05-27 13:55 [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin @ 2026-05-27 13:57 ` wang.yaxin 2026-05-27 13:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin ` (2 subsequent siblings) 3 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-05-27 13:57 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Previously delaytop only showed average delays. Add delay_max fields to track the maximum delay value for each delay type (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) per task. This provides a global view of all tasks' delay spikes, which is essential for identifying processes that experienced brief but significant latency events that would be hidden by average-only metrics. The -t/--type option displays only the specified delay type with avg/max values side by side, allowing focused analysis: delaytop -t cpu # Show only CPU delay with avg/max delaytop -t wpcopy # Show Copy-on-Write delay with avg/max Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 142 +++++++++++++++++++++++++++++------- 1 file changed, 117 insertions(+), 25 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 72cc500b44b1..586294b29834 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -82,6 +82,7 @@ #define MODE_TYPE_ALL (0xFFFFFFFF) #define MODE_DEFAULT (1 << 0) #define MODE_MEMVERBOSE (1 << 1) +#define MODE_TYPE (1 << 2) /* Display specific type with avg/max */ /* PSI statistics structure */ struct psi_stats { @@ -108,20 +109,28 @@ struct task_info { char command[TASK_COMM_LEN]; unsigned long long cpu_count; unsigned long long cpu_delay_total; + unsigned long long cpu_delay_max; unsigned long long blkio_count; unsigned long long blkio_delay_total; + unsigned long long blkio_delay_max; unsigned long long swapin_count; unsigned long long swapin_delay_total; + unsigned long long swapin_delay_max; unsigned long long freepages_count; unsigned long long freepages_delay_total; + unsigned long long freepages_delay_max; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; + unsigned long long thrashing_delay_max; unsigned long long compact_count; unsigned long long compact_delay_total; + unsigned long long compact_delay_max; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; + unsigned long long wpcopy_delay_max; unsigned long long irq_count; unsigned long long irq_delay_total; + unsigned long long irq_delay_max; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -153,6 +162,7 @@ struct config { int monitor_pid; /* Monitor specific PID */ char *container_path; /* Path to container cgroup */ const struct field_desc *sort_field; /* Current sort field */ + const struct field_desc *type_field; /* Type field for -t option */ size_t display_mode; /* Current display mode */ }; @@ -164,15 +174,15 @@ static int task_count; static int running = 1; static struct container_stats container_stats; static const struct field_desc sort_fields[] = { - SORT_FIELD(cpu, c, MODE_DEFAULT), - SORT_FIELD(blkio, i, MODE_DEFAULT), - SORT_FIELD(irq, q, MODE_DEFAULT), + SORT_FIELD(cpu, c, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(blkio, i, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(irq, q, MODE_DEFAULT | MODE_TYPE), SORT_FIELD(mem, m, MODE_DEFAULT | MODE_MEMVERBOSE), - SORT_FIELD(swapin, s, MODE_MEMVERBOSE), - SORT_FIELD(freepages, r, MODE_MEMVERBOSE), - SORT_FIELD(thrashing, t, MODE_MEMVERBOSE), - SORT_FIELD(compact, p, MODE_MEMVERBOSE), - SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE), + SORT_FIELD(swapin, s, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(freepages, r, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(thrashing, t, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(compact, p, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE | MODE_TYPE), END_FIELD }; static int sort_selected; @@ -218,7 +228,7 @@ static const struct field_desc *get_field_by_name(const char *name) for (field = sort_fields; field->name != NULL; field++) { field_len = strlen(field->name); - if (field_len != strlen(name)) + if (field_len != strlen(name) || !(field->supported_modes & MODE_TYPE)) continue; if (strncmp(field->name, name, field_len) == 0) return field; @@ -265,6 +275,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" + " -t, --type=FIELD Display only specified delay type with avg/max\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -283,6 +294,7 @@ static void parse_args(int argc, char **argv) {"processes", required_argument, 0, 'P'}, {"sort", required_argument, 0, 's'}, {"container", required_argument, 0, 'C'}, + {"type", required_argument, 0, 't'}, {"memverbose", no_argument, 0, 'M'}, {0, 0, 0, 0} }; @@ -292,6 +304,7 @@ static void parse_args(int argc, char **argv) cfg.iterations = 0; cfg.max_processes = 20; cfg.sort_field = &sort_fields[0]; /* Default sorted by CPU delay */ + cfg.type_field = NULL; /* No type field by default */ cfg.output_one_time = 0; cfg.monitor_pid = 0; /* 0 means monitor all PIDs */ cfg.container_path = NULL; @@ -300,7 +313,7 @@ static void parse_args(int argc, char **argv) while (1) { int option_index = 0; - c = getopt_long(argc, argv, "hd:n:p:oP:C:s:M", long_options, &option_index); + c = getopt_long(argc, argv, "hd:n:p:oP:C:s:t:M", long_options, &option_index); if (c == -1) break; @@ -363,6 +376,23 @@ static void parse_args(int argc, char **argv) cfg.sort_field = field; break; + case 't': + if (strlen(optarg) == 0) { + fprintf(stderr, "Error: empty type field\n"); + exit(1); + } + + field = get_field_by_name(optarg); + /* Show available fields if invalid option provided */ + if (!field) { + fprintf(stderr, "Error: invalid type field '%s'\n", optarg); + display_available_fields(MODE_TYPE); + exit(1); + } + + cfg.type_field = field; + cfg.display_mode = MODE_TYPE; + break; case 'M': cfg.display_mode = MODE_MEMVERBOSE; cfg.sort_field = get_field_by_name("mem"); @@ -699,20 +729,28 @@ static void fetch_and_fill_task_info(int pid, const char *comm) tasks[task_count].command[TASK_COMM_LEN - 1] = '\0'; SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); + SET_TASK_STAT(task_count, cpu_delay_max); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); + SET_TASK_STAT(task_count, blkio_delay_max); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); + SET_TASK_STAT(task_count, swapin_delay_max); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); + SET_TASK_STAT(task_count, freepages_delay_max); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); + SET_TASK_STAT(task_count, thrashing_delay_max); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); + SET_TASK_STAT(task_count, compact_delay_max); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); + SET_TASK_STAT(task_count, wpcopy_delay_max); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); + SET_TASK_STAT(task_count, irq_delay_max); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -794,6 +832,45 @@ static int compare_tasks(const void *a, const void *b) return 0; } +/* Get delay values for a specific field */ +static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, + double *avg_ms, double *max_ms) +{ + unsigned long long total, count, max; + + if (!field) { + *avg_ms = 0; + *max_ms = 0; + return; + } + + total = *(unsigned long long *)((char *)task + field->total_offset); + count = *(unsigned long *)((char *)task + field->count_offset); + *avg_ms = average_ms(total, count); + + /* Get max delay based on field name */ + if (strcmp(field->name, "cpu") == 0) + max = task->cpu_delay_max; + else if (strcmp(field->name, "blkio") == 0) + max = task->blkio_delay_max; + else if (strcmp(field->name, "irq") == 0) + max = task->irq_delay_max; + else if (strcmp(field->name, "swapin") == 0) + max = task->swapin_delay_max; + else if (strcmp(field->name, "freepages") == 0) + max = task->freepages_delay_max; + else if (strcmp(field->name, "thrashing") == 0) + max = task->thrashing_delay_max; + else if (strcmp(field->name, "compact") == 0) + max = task->compact_delay_max; + else if (strcmp(field->name, "wpcopy") == 0) + max = task->wpcopy_delay_max; + else + max = 0; + + *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ +} + /* Sort tasks by selected field */ static void sort_tasks(void) { @@ -950,21 +1027,29 @@ static void display_results(int psi_ret) suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", cfg.max_processes, get_name_by_field(cfg.sort_field)); - suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); - if (cfg.display_mode == MODE_MEMVERBOSE) { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", - "MEM(ms)", "SWAP(ms)", "RCL(ms)", - "THR(ms)", "CMP(ms)", "WP(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "---------------------\n"); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + /* Display mode for -t option: show only specified type with avg/max */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + "PID", "TGID", "COMMAND", + "AVG(ms)", "MAX(ms)"); + suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); } else { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", - "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "--------------------------\n"); + suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); + if (cfg.display_mode == MODE_MEMVERBOSE) { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", + "MEM(ms)", "SWAP(ms)", "RCL(ms)", + "THR(ms)", "CMP(ms)", "WP(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "---------------------\n"); + } else { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", + "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "--------------------------\n"); + } } count = task_count < cfg.max_processes ? task_count : cfg.max_processes; @@ -972,7 +1057,14 @@ static void display_results(int psi_ret) for (i = 0; i < count; i++) { suc &= BOOL_FPRINT(out, "%8d %8d %-15s", tasks[i].pid, tasks[i].tgid, tasks[i].command); - if (cfg.display_mode == MODE_MEMVERBOSE) { + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + double avg_ms, max_ms; + + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + + suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", + avg_ms, max_ms); + } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), TASK_AVG(tasks[i], swapin), -- 2.25.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/3] delaytop: add timestamp of delay max 2026-05-27 13:55 [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-05-27 13:57 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin @ 2026-05-27 13:58 ` wang.yaxin 2026-05-27 13:59 ` [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes wang.yaxin 2026-05-27 19:42 ` [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis Andrew Morton 3 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-05-27 13:58 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Record the wall-clock timestamp when each maximum delay occurred for all delay types. The timestamp is displayed in the MAX_TIMESTAMP column when using -t/--type option. This enables: - Identifying the time when a process experienced an abnormal delay spike - Correlating delay peaks across multiple processes at the same timestamp - Cross-referencing with system logs, traces, or other metrics at that time - Pinpointing the root cause of latency issues by finding concurrent events Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 100 ++++++++++++++++++++++++++++-------- 1 file changed, 80 insertions(+), 20 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 586294b29834..8d22c43dceed 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -110,27 +110,35 @@ struct task_info { unsigned long long cpu_count; unsigned long long cpu_delay_total; unsigned long long cpu_delay_max; + struct __kernel_timespec cpu_delay_max_ts; unsigned long long blkio_count; unsigned long long blkio_delay_total; unsigned long long blkio_delay_max; + struct __kernel_timespec blkio_delay_max_ts; unsigned long long swapin_count; unsigned long long swapin_delay_total; unsigned long long swapin_delay_max; + struct __kernel_timespec swapin_delay_max_ts; unsigned long long freepages_count; unsigned long long freepages_delay_total; unsigned long long freepages_delay_max; + struct __kernel_timespec freepages_delay_max_ts; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; unsigned long long thrashing_delay_max; + struct __kernel_timespec thrashing_delay_max_ts; unsigned long long compact_count; unsigned long long compact_delay_total; unsigned long long compact_delay_max; + struct __kernel_timespec compact_delay_max_ts; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; unsigned long long wpcopy_delay_max; + struct __kernel_timespec wpcopy_delay_max_ts; unsigned long long irq_count; unsigned long long irq_delay_total; unsigned long long irq_delay_max; + struct __kernel_timespec irq_delay_max_ts; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -275,7 +283,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" - " -t, --type=FIELD Display only specified delay type with avg/max\n" + " -t, --type=FIELD Display only specified delay type with avg/max/timestamp\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -730,27 +738,34 @@ static void fetch_and_fill_task_info(int pid, const char *comm) SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); SET_TASK_STAT(task_count, cpu_delay_max); + SET_TASK_STAT(task_count, cpu_delay_max_ts); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); SET_TASK_STAT(task_count, blkio_delay_max); + SET_TASK_STAT(task_count, blkio_delay_max_ts); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); SET_TASK_STAT(task_count, swapin_delay_max); + SET_TASK_STAT(task_count, swapin_delay_max_ts); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); SET_TASK_STAT(task_count, freepages_delay_max); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); SET_TASK_STAT(task_count, thrashing_delay_max); + SET_TASK_STAT(task_count, thrashing_delay_max_ts); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); SET_TASK_STAT(task_count, compact_delay_max); + SET_TASK_STAT(task_count, compact_delay_max_ts); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); SET_TASK_STAT(task_count, wpcopy_delay_max); + SET_TASK_STAT(task_count, wpcopy_delay_max_ts); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); SET_TASK_STAT(task_count, irq_delay_max); + SET_TASK_STAT(task_count, irq_delay_max_ts); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -808,6 +823,38 @@ static double average_ms(unsigned long long total, unsigned long long count) return (double)total / 1000000.0 / count; } +/* + * Format __kernel_timespec to human readable string (YYYY-MM-DD HH:MM:SS) + * Returns formatted string or "N/A" if timestamp is zero + */ +static const char *format_timespec64(struct __kernel_timespec *ts) +{ + static char buffer[32]; + struct tm *tm_info; + time_t time_sec; + + /* Check if timestamp is zero (not set) or invalid (before year 2000) */ + if ((ts->tv_sec == 0 && ts->tv_nsec == 0) || ts->tv_sec < 946684800) { + /* 946684800 is timestamp for 2000-01-01 00:00:00 UTC */ + return "N/A"; + } + + time_sec = (time_t)ts->tv_sec; + tm_info = localtime(&time_sec); + if (!tm_info) + return "N/A"; + + snprintf(buffer, sizeof(buffer), "%04d-%02d-%02d %02d:%02d:%02d", + tm_info->tm_year + 1900, + tm_info->tm_mon + 1, + tm_info->tm_mday, + tm_info->tm_hour, + tm_info->tm_min, + tm_info->tm_sec); + + return buffer; +} + /* Comparison function for sorting tasks */ static int compare_tasks(const void *a, const void *b) { @@ -834,13 +881,14 @@ static int compare_tasks(const void *a, const void *b) /* Get delay values for a specific field */ static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, - double *avg_ms, double *max_ms) + double *avg_ms, double *max_ms, struct __kernel_timespec *max_ts) { unsigned long long total, count, max; if (!field) { *avg_ms = 0; *max_ms = 0; + memset(max_ts, 0, sizeof(*max_ts)); return; } @@ -848,26 +896,35 @@ static void get_field_delay_values(const struct task_info *task, const struct fi count = *(unsigned long *)((char *)task + field->count_offset); *avg_ms = average_ms(total, count); - /* Get max delay based on field name */ - if (strcmp(field->name, "cpu") == 0) + /* Get max delay and timestamp based on field name */ + if (strcmp(field->name, "cpu") == 0) { max = task->cpu_delay_max; - else if (strcmp(field->name, "blkio") == 0) + *max_ts = task->cpu_delay_max_ts; + } else if (strcmp(field->name, "blkio") == 0) { max = task->blkio_delay_max; - else if (strcmp(field->name, "irq") == 0) + *max_ts = task->blkio_delay_max_ts; + } else if (strcmp(field->name, "irq") == 0) { max = task->irq_delay_max; - else if (strcmp(field->name, "swapin") == 0) + *max_ts = task->irq_delay_max_ts; + } else if (strcmp(field->name, "swapin") == 0) { max = task->swapin_delay_max; - else if (strcmp(field->name, "freepages") == 0) + *max_ts = task->swapin_delay_max_ts; + } else if (strcmp(field->name, "freepages") == 0) { max = task->freepages_delay_max; - else if (strcmp(field->name, "thrashing") == 0) + *max_ts = task->freepages_delay_max_ts; + } else if (strcmp(field->name, "thrashing") == 0) { max = task->thrashing_delay_max; - else if (strcmp(field->name, "compact") == 0) + *max_ts = task->thrashing_delay_max_ts; + } else if (strcmp(field->name, "compact") == 0) { max = task->compact_delay_max; - else if (strcmp(field->name, "wpcopy") == 0) + *max_ts = task->compact_delay_max_ts; + } else if (strcmp(field->name, "wpcopy") == 0) { max = task->wpcopy_delay_max; - else + *max_ts = task->wpcopy_delay_max_ts; + } else { max = 0; - + memset(max_ts, 0, sizeof(*max_ts)); + } *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ } @@ -1028,11 +1085,12 @@ static void display_results(int psi_ret) cfg.max_processes, get_name_by_field(cfg.sort_field)); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { - /* Display mode for -t option: show only specified type with avg/max */ - suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + /* Display mode for -t option: show only specified type with avg/max/timestamp */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s %20s\n", "PID", "TGID", "COMMAND", - "AVG(ms)", "MAX(ms)"); - suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); + "AVG(ms)", "MAX(ms)", "MAX_TIMESTAMP"); + suc &= BOOL_FPRINT(out, "--------------------------------------------------------"); + suc &= BOOL_FPRINT(out, "----------------------------------------\n"); } else { suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); if (cfg.display_mode == MODE_MEMVERBOSE) { @@ -1059,11 +1117,13 @@ static void display_results(int psi_ret) tasks[i].pid, tasks[i].tgid, tasks[i].command); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { double avg_ms, max_ms; + struct __kernel_timespec max_ts; - get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, + &max_ms, &max_ts); - suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", - avg_ms, max_ms); + suc &= BOOL_FPRINT(out, "%12.2f %12.2f %20s\n", + avg_ms, max_ms, format_timespec64(&max_ts)); } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), -- 2.25.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes 2026-05-27 13:55 [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-05-27 13:57 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin 2026-05-27 13:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin @ 2026-05-27 13:59 ` wang.yaxin 2026-05-27 19:42 ` [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis Andrew Morton 3 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-05-27 13:59 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> When using -t/--type option, sort tasks by the maximum delay value of the selected type in descending order (largest delay first). This enables quickly identifying the top N processes with the highest delay spikes, which is essential for diagnosing latency problems by pinpointing which processes contributed most to system delays. Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- Documentation/accounting/delay-accounting.rst | 43 +++++ tools/accounting/delaytop.c | 162 ++++++++++-------- 2 files changed, 136 insertions(+), 69 deletions(-) diff --git a/Documentation/accounting/delay-accounting.rst b/Documentation/accounting/delay-accounting.rst index e209c46241b0..a42d830e65ce 100644 --- a/Documentation/accounting/delay-accounting.rst +++ b/Documentation/accounting/delay-accounting.rst @@ -212,3 +212,46 @@ Advanced usage examples:: # ./delaytop -d secs Specify refresh interval as secs + + # ./delaytop -t type + Display only specified delay type with avg/max/timestamp + (rows sorted by MAX for that type, largest first) + + + +delaytop add delay_max fields to track the maximum delay value for each delay type +(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) per task. + + +System Pressure Information: (avg10/avg60vg300/total) +CPU some: 0.4%/ 0.2%/ 0.1%/ 220(ms) +CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) +Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) +Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) +IO full: 0.0%/ 0.0%/ 0.0%/ 12(ms) +IO some: 0.0%/ 0.0%/ 0.0%/ 13(ms) +IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) +[o]sort [M]memverbose [q]quit +Top 20 processes (sorted by cpu MAX delay, largest first): + PID TGID COMMAND AVG(ms) MAX(ms) MAX_TIMESTAMP +------------------------------------------------------------------------------------------------ + 9 9 kworker/0:0-eve 0.59 16.87 2026-05-27T13:32:39 + 30 30 kworker/2:0H-kb 2.87 11.36 2026-05-27T13:32:36 + 27 27 migration/2 1.05 9.51 2026-05-27T13:32:37 + 50 50 kworker/2:1-eve 0.50 9.13 2026-05-27T13:32:37 + 15 15 rcu_preempt 0.11 8.98 2026-05-27T13:32:37 + 1 1 init 0.17 7.12 2026-05-27T13:32:38 + 67 67 scsi_eh_0 1.20 4.23 2026-05-27T13:32:37 + 23 23 ksoftirqd/1 1.12 3.77 2026-05-27T13:32:36 + 3 3 pool_workqueue_ 0.72 3.55 2026-05-27T13:32:38 + 62 62 kworker/u20:2-a 0.49 3.03 2026-05-27T13:32:37 + 2 2 kthreadd 0.18 2.82 2026-05-27T13:32:37 + 11 11 kworker/0:1 1.42 2.76 2026-05-27T13:32:36 + 39 39 kworker/u20:0-a 0.10 2.71 2026-05-27T13:32:38 + 17 17 rcu_exp_gp_kthr 0.25 2.65 2026-05-27T13:32:37 + 66 66 kworker/u20:3-e 0.38 2.55 2026-05-27T13:32:37 + 20 20 cpuhp/0 0.53 2.51 2026-05-27T13:32:37 + 28 28 ksoftirqd/2 0.59 2.48 2026-05-27T13:32:37 + 55 55 kworker/u19:1 0.88 2.42 2026-05-27T13:32:37 + 13 13 kworker/R-mm_pe 1.18 2.35 2026-05-27T13:32:36 + 54 54 kworker/3:1-eve 0.14 2.20 2026-05-27T13:32:38 diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 8d22c43dceed..87fb45a6a6cd 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -284,6 +284,7 @@ static void usage(void) " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" " -t, --type=FIELD Display only specified delay type with avg/max/timestamp\n" + " (rows sorted by MAX for that type, largest first)\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -823,111 +824,129 @@ static double average_ms(unsigned long long total, unsigned long long count) return (double)total / 1000000.0 / count; } -/* - * Format __kernel_timespec to human readable string (YYYY-MM-DD HH:MM:SS) - * Returns formatted string or "N/A" if timestamp is zero - */ -static const char *format_timespec64(struct __kernel_timespec *ts) +static const char *format_kernel_timespec(struct __kernel_timespec *ts) { static char buffer[32]; - struct tm *tm_info; - time_t time_sec; + struct tm tm_info; + __kernel_time_t time_sec; - /* Check if timestamp is zero (not set) or invalid (before year 2000) */ - if ((ts->tv_sec == 0 && ts->tv_nsec == 0) || ts->tv_sec < 946684800) { - /* 946684800 is timestamp for 2000-01-01 00:00:00 UTC */ + /* Check if timestamp is zero (not set) */ + if (ts->tv_sec == 0 && ts->tv_nsec == 0) return "N/A"; - } - time_sec = (time_t)ts->tv_sec; - tm_info = localtime(&time_sec); - if (!tm_info) + time_sec = ts->tv_sec; + + /* Use thread-safe localtime_r */ + if (localtime_r(&time_sec, &tm_info) == NULL) return "N/A"; - snprintf(buffer, sizeof(buffer), "%04d-%02d-%02d %02d:%02d:%02d", - tm_info->tm_year + 1900, - tm_info->tm_mon + 1, - tm_info->tm_mday, - tm_info->tm_hour, - tm_info->tm_min, - tm_info->tm_sec); + snprintf(buffer, sizeof(buffer), "%04d-%02d-%02dT%02d:%02d:%02d", + tm_info.tm_year + 1900, + tm_info.tm_mon + 1, + tm_info.tm_mday, + tm_info.tm_hour, + tm_info.tm_min, + tm_info.tm_sec); return buffer; } -/* Comparison function for sorting tasks */ -static int compare_tasks(const void *a, const void *b) -{ - const struct task_info *t1 = (const struct task_info *)a; - const struct task_info *t2 = (const struct task_info *)b; - unsigned long long total1; - unsigned long long total2; - unsigned long count1; - unsigned long count2; - double avg1, avg2; - - total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); - total2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->total_offset); - count1 = *(unsigned long *)((char *)t1 + cfg.sort_field->count_offset); - count2 = *(unsigned long *)((char *)t2 + cfg.sort_field->count_offset); - - avg1 = average_ms(total1, count1); - avg2 = average_ms(total2, count2); - if (avg1 != avg2) - return avg2 > avg1 ? 1 : -1; - - return 0; -} - -/* Get delay values for a specific field */ -static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, - double *avg_ms, double *max_ms, struct __kernel_timespec *max_ts) +/* Max delay (ns) and timestamp for field (shared by display and sort) */ +void field_delay_max_and_ts(const struct task_info *task, const struct field_desc *field, + unsigned long long *max_ns, struct __kernel_timespec *max_ts) { - unsigned long long total, count, max; - if (!field) { - *avg_ms = 0; - *max_ms = 0; + *max_ns = 0; memset(max_ts, 0, sizeof(*max_ts)); return; } - total = *(unsigned long long *)((char *)task + field->total_offset); - count = *(unsigned long *)((char *)task + field->count_offset); - *avg_ms = average_ms(total, count); - - /* Get max delay and timestamp based on field name */ if (strcmp(field->name, "cpu") == 0) { - max = task->cpu_delay_max; + *max_ns = task->cpu_delay_max; *max_ts = task->cpu_delay_max_ts; } else if (strcmp(field->name, "blkio") == 0) { - max = task->blkio_delay_max; + *max_ns = task->blkio_delay_max; *max_ts = task->blkio_delay_max_ts; } else if (strcmp(field->name, "irq") == 0) { - max = task->irq_delay_max; + *max_ns = task->irq_delay_max; *max_ts = task->irq_delay_max_ts; } else if (strcmp(field->name, "swapin") == 0) { - max = task->swapin_delay_max; + *max_ns = task->swapin_delay_max; *max_ts = task->swapin_delay_max_ts; } else if (strcmp(field->name, "freepages") == 0) { - max = task->freepages_delay_max; + *max_ns = task->freepages_delay_max; *max_ts = task->freepages_delay_max_ts; } else if (strcmp(field->name, "thrashing") == 0) { - max = task->thrashing_delay_max; + *max_ns = task->thrashing_delay_max; *max_ts = task->thrashing_delay_max_ts; } else if (strcmp(field->name, "compact") == 0) { - max = task->compact_delay_max; + *max_ns = task->compact_delay_max; *max_ts = task->compact_delay_max_ts; } else if (strcmp(field->name, "wpcopy") == 0) { - max = task->wpcopy_delay_max; + *max_ns = task->wpcopy_delay_max; *max_ts = task->wpcopy_delay_max_ts; } else { - max = 0; + *max_ns = 0; + memset(max_ts, 0, sizeof(*max_ts)); + } +} + +/* Get delay values for a specific field */ +void get_field_delay_values(const struct task_info *task, const struct field_desc *field, + double *avg_ms, double *max_ms, struct __kernel_timespec *max_ts) +{ + unsigned long long total, count, max; + + if (!field) { + *avg_ms = 0; + *max_ms = 0; memset(max_ts, 0, sizeof(*max_ts)); + return; } + + total = *(unsigned long long *)((char *)task + field->total_offset); + count = *(unsigned long *)((char *)task + field->count_offset); + *avg_ms = average_ms(total, count); + + field_delay_max_and_ts(task, field, &max, max_ts); *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ } +/* Comparison function for sorting tasks */ +static int compare_tasks(const void *a, const void *b) +{ + const struct task_info *t1 = (const struct task_info *)a; + const struct task_info *t2 = (const struct task_info *)b; + unsigned long long total1; + unsigned long long total2; + unsigned long count1; + unsigned long count2; + double avg1, avg2; + unsigned long long max1, max2; + struct __kernel_timespec ts_scratch; + + /* -t/--type: default sort by MAX column for the selected type (descending) */ + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + field_delay_max_and_ts(t1, cfg.type_field, &max1, &ts_scratch); + field_delay_max_and_ts(t2, cfg.type_field, &max2, &ts_scratch); + if (max1 != max2) + return max2 > max1 ? 1 : -1; + return 0; + } + + total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); + total2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->total_offset); + count1 = *(unsigned long *)((char *)t1 + cfg.sort_field->count_offset); + count2 = *(unsigned long *)((char *)t2 + cfg.sort_field->count_offset); + + avg1 = average_ms(total1, count1); + avg2 = average_ms(total2, count2); + if (avg1 != avg2) + return avg2 > avg1 ? 1 : -1; + + return 0; +} + /* Sort tasks by selected field */ static void sort_tasks(void) { @@ -1081,8 +1100,13 @@ static void display_results(int psi_ret) } /* Task delay output */ - suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", - cfg.max_processes, get_name_by_field(cfg.sort_field)); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) + suc &= BOOL_FPRINT(out, + "Top %d processes (sorted by %s MAX delay, largest first):\n", + cfg.max_processes, get_name_by_field(cfg.type_field)); + else + suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", + cfg.max_processes, get_name_by_field(cfg.sort_field)); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { /* Display mode for -t option: show only specified type with avg/max/timestamp */ @@ -1123,7 +1147,7 @@ static void display_results(int psi_ret) &max_ms, &max_ts); suc &= BOOL_FPRINT(out, "%12.2f %12.2f %20s\n", - avg_ms, max_ms, format_timespec64(&max_ts)); + avg_ms, max_ms, format_kernel_timespec(&max_ts)); } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), -- 2.25.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis 2026-05-27 13:55 [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin ` (2 preceding siblings ...) 2026-05-27 13:59 ` [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes wang.yaxin @ 2026-05-27 19:42 ` Andrew Morton 2026-05-28 4:38 ` wang.yaxin 3 siblings, 1 reply; 9+ messages in thread From: Andrew Morton @ 2026-05-27 19:42 UTC (permalink / raw) To: wang.yaxin Cc: fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 On Wed, 27 May 2026 21:55:24 +0800 (CST) <wang.yaxin@zte.com.cn> wrote: > From: Wang Yaxin <wang.yaxin@zte.com.cn> > > Previously delaytop only showed average delays. This patch adds: > > 1. delay_max fields to track the maximum delay value for each delay type > (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) > per task. > > 2. The -t/--type option displays only the specified delay type with avg/max > values side by side, allowing focused analysis: > delaytop -t cpu # Show only CPU delay with avg/max > delaytop -t wpcopy # Show Copy-on-Write delay with avg/max > > 3. Wall-clock timestamp when each maximum delay occurred, displayed in the > MAX_TIMESTAMP column when using -t/--type option. This enables: > - Identifying the time when a process experienced an abnormal delay max > - Correlating delay max across multiple processes at the same timestamp > - Cross-referencing with logs, traces, or other metrics at that time > > 4. When using -t/--type option, tasks are sorted by maximum delay value in > descending order (largest delay first), enabling quick identification of > top N processes with highest delay spikes. Sounds useful. Am I correct in assuming that you're a regular user of delaytop and that these changes are based on your experience with it? AI review claims to have found a number of things: https://sashiko.dev/#/patchset/20260527215524044fG7XSpgveHiaFhraq0yAi@zte.com.cn ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis 2026-05-27 19:42 ` [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis Andrew Morton @ 2026-05-28 4:38 ` wang.yaxin 0 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-05-28 4:38 UTC (permalink / raw) To: akpm; +Cc: fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 >> From: Wang Yaxin <wang.yaxin@zte.com.cn> >> >> Previously delaytop only showed average delays. This patch adds: >> >> 1. delay_max fields to track the maximum delay value for each delay type >> (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) >> per task. >> >> 2. The -t/--type option displays only the specified delay type with avg/max >> values side by side, allowing focused analysis: >> delaytop -t cpu # Show only CPU delay with avg/max >> delaytop -t wpcopy # Show Copy-on-Write delay with avg/max >> >> 3. Wall-clock timestamp when each maximum delay occurred, displayed in the >> MAX_TIMESTAMP column when using -t/--type option. This enables: >> - Identifying the time when a process experienced an abnormal delay max >> - Correlating delay max across multiple processes at the same timestamp >> - Cross-referencing with logs, traces, or other metrics at that time >> >> 4. When using -t/--type option, tasks are sorted by maximum delay value in >> descending order (largest delay first), enabling quick identification of >> top N processes with highest delay spikes. > >Sounds useful. Am I correct in assuming that you're a regular user of >delaytop and that these changes are based on your experience with it? This indeed comes from latency observation requirements in business scenarios. Some of our tasks are highly latency-sensitive, so we need to monitor the "delay max" of specific tasks and detect anomalies in a timely manner. delaytop is very useful for observing task-level latency. Thanks for the review. I will submit the v2 patch to fix the potential issues as suggested Thanks Yaxin ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 0/3 v2] delaytop: add delay max, timestamp and sorting for top latency analysis
@ 2026-05-30 3:16 wang.yaxin
2026-05-30 3:17 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin
0 siblings, 1 reply; 9+ messages in thread
From: wang.yaxin @ 2026-05-30 3:16 UTC (permalink / raw)
To: akpm, fan.yu9, yang.yang29
Cc: corbet, linux-kernel, linux-doc, xu.xin16, wang.yaxin
From: Wang Yaxin <wang.yaxin@zte.com.cn>
Previously delaytop only showed average delays. This patch adds:
1. delay_max fields to track the maximum delay value for each delay type
(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
per task.
2. The -t/--type option displays only the specified delay type with avg/max
values side by side, allowing focused analysis:
delaytop -t cpu # Show only CPU delay with avg/max
delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
3. Wall-clock timestamp when each maximum delay occurred, displayed in the
MAX_TIMESTAMP column when using -t/--type option. This enables:
- Identifying the time when a process experienced an abnormal delay max
- Correlating delay max across multiple processes at the same timestamp
- Cross-referencing with logs, traces, or other metrics at that time
4. When using -t/--type option, tasks are sorted by maximum delay value in
descending order (largest delay first), enabling quick identification of
top N processes with highest delay spikes.
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
v1->v2:
https://lore.kernel.org/all/20260527124210.19726c1c89a94b89310e5a47@linux-foundation.org/
[patch 1/3]
1. Fix potential NULL pointer dereference in get_field_by_name().
2. Fix data truncation by changing count pointer type from unsigned long to
unsigned long long.
3. Fix column misalignment by changing COMMAND format from %-15s to %-17s.
[patch 2/3]
1. Add missing freepages_delay_max_ts initialization to avoid stale
timestamps.
2. Fix Year 2038 regression by removing explicit time_t cast.
[patch 3/3]
1. Fix Y2038 truncation on 32-bit platforms by checking timestamp
overflow before cast.
2. Mark field_delay_max_and_ts and get_field_delay_values as static
to avoid missing-prototypes warnings.
3. Fix unexpected unindent in the Documentation.
Wang Yaxin (3):
delaytop: add delay max for delaytop
delaytop: add timestamp of delay max
delaytop: sort by max delay to highlight top latency processes
Documentation/accounting/delay-accounting.rst | 43 +++
tools/accounting/delaytop.c | 263 ++++++++++++++++--
2 files changed, 276 insertions(+), 30 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 1/3] delaytop: add delay max for delaytop 2026-05-30 3:16 [PATCH 0/3 v2] " wang.yaxin @ 2026-05-30 3:17 ` wang.yaxin 0 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-05-30 3:17 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Previously delaytop only showed average delays. Add delay_max fields to track the maximum delay value for each delay type (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) per task. This provides a global view of all tasks' delay spikes, which is essential for identifying processes that experienced brief but significant latency events that would be hidden by average-only metrics. The -t/--type option displays only the specified delay type with avg/max values side by side, allowing focused analysis: delaytop -t cpu # Show only CPU delay with avg/max delaytop -t wpcopy # Show Copy-on-Write delay with avg/max Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 152 +++++++++++++++++++++++++++++------- 1 file changed, 125 insertions(+), 27 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 72cc500b44b1..015ce8b6917f 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -82,6 +82,7 @@ #define MODE_TYPE_ALL (0xFFFFFFFF) #define MODE_DEFAULT (1 << 0) #define MODE_MEMVERBOSE (1 << 1) +#define MODE_TYPE (1 << 2) /* Display specific type with avg/max */ /* PSI statistics structure */ struct psi_stats { @@ -108,20 +109,28 @@ struct task_info { char command[TASK_COMM_LEN]; unsigned long long cpu_count; unsigned long long cpu_delay_total; + unsigned long long cpu_delay_max; unsigned long long blkio_count; unsigned long long blkio_delay_total; + unsigned long long blkio_delay_max; unsigned long long swapin_count; unsigned long long swapin_delay_total; + unsigned long long swapin_delay_max; unsigned long long freepages_count; unsigned long long freepages_delay_total; + unsigned long long freepages_delay_max; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; + unsigned long long thrashing_delay_max; unsigned long long compact_count; unsigned long long compact_delay_total; + unsigned long long compact_delay_max; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; + unsigned long long wpcopy_delay_max; unsigned long long irq_count; unsigned long long irq_delay_total; + unsigned long long irq_delay_max; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -153,6 +162,7 @@ struct config { int monitor_pid; /* Monitor specific PID */ char *container_path; /* Path to container cgroup */ const struct field_desc *sort_field; /* Current sort field */ + const struct field_desc *type_field; /* Type field for -t option */ size_t display_mode; /* Current display mode */ }; @@ -164,15 +174,15 @@ static int task_count; static int running = 1; static struct container_stats container_stats; static const struct field_desc sort_fields[] = { - SORT_FIELD(cpu, c, MODE_DEFAULT), - SORT_FIELD(blkio, i, MODE_DEFAULT), - SORT_FIELD(irq, q, MODE_DEFAULT), + SORT_FIELD(cpu, c, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(blkio, i, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(irq, q, MODE_DEFAULT | MODE_TYPE), SORT_FIELD(mem, m, MODE_DEFAULT | MODE_MEMVERBOSE), - SORT_FIELD(swapin, s, MODE_MEMVERBOSE), - SORT_FIELD(freepages, r, MODE_MEMVERBOSE), - SORT_FIELD(thrashing, t, MODE_MEMVERBOSE), - SORT_FIELD(compact, p, MODE_MEMVERBOSE), - SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE), + SORT_FIELD(swapin, s, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(freepages, r, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(thrashing, t, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(compact, p, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE | MODE_TYPE), END_FIELD }; static int sort_selected; @@ -218,7 +228,7 @@ static const struct field_desc *get_field_by_name(const char *name) for (field = sort_fields; field->name != NULL; field++) { field_len = strlen(field->name); - if (field_len != strlen(name)) + if (field_len != strlen(name) || !(field->supported_modes & MODE_TYPE)) continue; if (strncmp(field->name, name, field_len) == 0) return field; @@ -265,6 +275,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" + " -t, --type=FIELD Display only specified delay type with avg/max\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -283,6 +294,7 @@ static void parse_args(int argc, char **argv) {"processes", required_argument, 0, 'P'}, {"sort", required_argument, 0, 's'}, {"container", required_argument, 0, 'C'}, + {"type", required_argument, 0, 't'}, {"memverbose", no_argument, 0, 'M'}, {0, 0, 0, 0} }; @@ -292,6 +304,7 @@ static void parse_args(int argc, char **argv) cfg.iterations = 0; cfg.max_processes = 20; cfg.sort_field = &sort_fields[0]; /* Default sorted by CPU delay */ + cfg.type_field = NULL; /* No type field by default */ cfg.output_one_time = 0; cfg.monitor_pid = 0; /* 0 means monitor all PIDs */ cfg.container_path = NULL; @@ -300,7 +313,7 @@ static void parse_args(int argc, char **argv) while (1) { int option_index = 0; - c = getopt_long(argc, argv, "hd:n:p:oP:C:s:M", long_options, &option_index); + c = getopt_long(argc, argv, "hd:n:p:oP:C:s:t:M", long_options, &option_index); if (c == -1) break; @@ -363,9 +376,32 @@ static void parse_args(int argc, char **argv) cfg.sort_field = field; break; + case 't': + if (strlen(optarg) == 0) { + fprintf(stderr, "Error: empty type field\n"); + exit(1); + } + + field = get_field_by_name(optarg); + /* Show available fields if invalid option provided */ + if (!field) { + fprintf(stderr, "Error: invalid type field '%s'\n", optarg); + display_available_fields(MODE_TYPE); + exit(1); + } + + cfg.type_field = field; + cfg.display_mode = MODE_TYPE; + break; case 'M': cfg.display_mode = MODE_MEMVERBOSE; - cfg.sort_field = get_field_by_name("mem"); + /* Find first field supporting MODE_MEMVERBOSE for sorting */ + for (field = sort_fields; field->name != NULL; field++) { + if (field->supported_modes & MODE_MEMVERBOSE) { + cfg.sort_field = field; + break; + } + } break; default: fprintf(stderr, "Try 'delaytop --help' for more information.\n"); @@ -699,20 +735,28 @@ static void fetch_and_fill_task_info(int pid, const char *comm) tasks[task_count].command[TASK_COMM_LEN - 1] = '\0'; SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); + SET_TASK_STAT(task_count, cpu_delay_max); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); + SET_TASK_STAT(task_count, blkio_delay_max); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); + SET_TASK_STAT(task_count, swapin_delay_max); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); + SET_TASK_STAT(task_count, freepages_delay_max); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); + SET_TASK_STAT(task_count, thrashing_delay_max); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); + SET_TASK_STAT(task_count, compact_delay_max); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); + SET_TASK_STAT(task_count, wpcopy_delay_max); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); + SET_TASK_STAT(task_count, irq_delay_max); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -794,6 +838,45 @@ static int compare_tasks(const void *a, const void *b) return 0; } +/* Get delay values for a specific field */ +static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, + double *avg_ms, double *max_ms) +{ + unsigned long long total, count, max; + + if (!field) { + *avg_ms = 0; + *max_ms = 0; + return; + } + + total = *(unsigned long long *)((char *)task + field->total_offset); + count = *(unsigned long long *)((char *)task + field->count_offset); + *avg_ms = average_ms(total, count); + + /* Get max delay based on field name */ + if (strcmp(field->name, "cpu") == 0) + max = task->cpu_delay_max; + else if (strcmp(field->name, "blkio") == 0) + max = task->blkio_delay_max; + else if (strcmp(field->name, "irq") == 0) + max = task->irq_delay_max; + else if (strcmp(field->name, "swapin") == 0) + max = task->swapin_delay_max; + else if (strcmp(field->name, "freepages") == 0) + max = task->freepages_delay_max; + else if (strcmp(field->name, "thrashing") == 0) + max = task->thrashing_delay_max; + else if (strcmp(field->name, "compact") == 0) + max = task->compact_delay_max; + else if (strcmp(field->name, "wpcopy") == 0) + max = task->wpcopy_delay_max; + else + max = 0; + + *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ +} + /* Sort tasks by selected field */ static void sort_tasks(void) { @@ -950,29 +1033,44 @@ static void display_results(int psi_ret) suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", cfg.max_processes, get_name_by_field(cfg.sort_field)); - suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); - if (cfg.display_mode == MODE_MEMVERBOSE) { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", - "MEM(ms)", "SWAP(ms)", "RCL(ms)", - "THR(ms)", "CMP(ms)", "WP(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "---------------------\n"); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + /* Display mode for -t option: show only specified type with avg/max */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + "PID", "TGID", "COMMAND", + "AVG(ms)", "MAX(ms)"); + suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); } else { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", - "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "--------------------------\n"); + suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); + if (cfg.display_mode == MODE_MEMVERBOSE) { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", + "MEM(ms)", "SWAP(ms)", "RCL(ms)", + "THR(ms)", "CMP(ms)", "WP(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "---------------------\n"); + } else { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", + "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "--------------------------\n"); + } } count = task_count < cfg.max_processes ? task_count : cfg.max_processes; for (i = 0; i < count; i++) { - suc &= BOOL_FPRINT(out, "%8d %8d %-15s", + suc &= BOOL_FPRINT(out, "%8d %8d %-17s", tasks[i].pid, tasks[i].tgid, tasks[i].command); - if (cfg.display_mode == MODE_MEMVERBOSE) { + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + double avg_ms, max_ms; + + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + + suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", + avg_ms, max_ms); + } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), TASK_AVG(tasks[i], swapin), -- 2.25.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis
@ 2026-07-02 12:57 wang.yaxin
2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin
0 siblings, 1 reply; 9+ messages in thread
From: wang.yaxin @ 2026-07-02 12:57 UTC (permalink / raw)
To: akpm, fan.yu9, yang.yang29
Cc: corbet, linux-kernel, linux-doc, xu.xin16, wang.yaxin
From: Wang Yaxin <wang.yaxin@zte.com.cn>
Previously delaytop only showed average delays. This patch adds:
1. delay_max fields to track the maximum delay value for each delay type
(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
per task.
2. The -t/--type option displays only the specified delay type with avg/max
values side by side, allowing focused analysis:
delaytop -t cpu # Show only CPU delay with avg/max
delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
3. Wall-clock timestamp when each maximum delay occurred, displayed in the
MAX_TIMESTAMP column when using -t/--type option. This enables:
- Identifying the time when a process experienced an abnormal delay max
- Correlating delay max across multiple processes at the same timestamp
- Cross-referencing with logs, traces, or other metrics at that time
4. When using -t/--type option, tasks are sorted by maximum delay value in
descending order (largest delay first), enabling quick identification of
top N processes with highest delay spikes.
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
v2->v3:
[patch 1/3]
1. fix get_field_by_name() MODE_TYPE filter breaking --sort for non-type fields
2. zero-init stats struct and cap memcpy size in netlink response parsing
3. fix compare_tasks() count field read as unsigned long long for 32-bit safety
4. strcmp → offset-based access via field_desc.max_offset
[patch 2/3]
1. fix 32-bit time_t overflow check to use max representable value instead of (time_t)-1
2. fix gmtime() pointer aliasing by assigning timespec64 tv_sec to local time_t
3. dropped pre-2000 check; gmtime → localtime_r + stack struct tm
[patch 3/3]
1. fixed Y2038 overflow false negative in format_kernel_timespec()
2. switched to localtime_r() with stack `struct tm` for thread safety.
3. `field_delay_max_and_ts()` `max_ts` nullable; removed unused ts_scratch
4. `-t/--type` mode: hide `[o]sort` and `[M]memverbose`, show only `[q]quit`
Wang Yaxin (3):
delaytop: add delay max for delaytop
delaytop: add timestamp of delay max
delaytop: sort by max delay to highlight top latency processes
Documentation/accounting/delay-accounting.rst | 43 +++
tools/accounting/delaytop.c | 286 +++++++++++++++---
tools/accounting/getdelays.c | 5 +
3 files changed, 296 insertions(+), 38 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 1/3] delaytop: add delay max for delaytop 2026-07-02 12:57 [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin @ 2026-07-02 12:58 ` wang.yaxin 2026-07-02 22:58 ` Andrew Morton 0 siblings, 1 reply; 9+ messages in thread From: wang.yaxin @ 2026-07-02 12:58 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Previously delaytop only showed average delays. Add delay_max fields to track the maximum delay value for each delay type (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) per task. This provides a global view of all tasks' delay spikes, which is essential for identifying processes that experienced brief but significant latency events that would be hidden by average-only metrics. The -t/--type option displays only the specified delay type with avg/max values side by side, allowing focused analysis: delaytop -t cpu # Show only CPU delay with avg/max delaytop -t wpcopy # Show Copy-on-Write delay with avg/max Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 165 ++++++++++++++++++++++++++++-------- 1 file changed, 131 insertions(+), 34 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 72cc500b44b1..4410f8add3e4 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -75,13 +75,21 @@ {#name, #cmd, \ offsetof(struct task_info, name##_delay_total), \ offsetof(struct task_info, name##_count), \ + offsetof(struct task_info, name##_delay_max), \ modes} -#define END_FIELD {NULL, 0, 0} +#define SORT_FIELD_NO_MAX(name, cmd, modes) \ + {#name, #cmd, \ + offsetof(struct task_info, name##_delay_total), \ + offsetof(struct task_info, name##_count), \ + 0, \ + modes} +#define END_FIELD {NULL, 0, 0, 0, 0, 0} /* Display mode types */ #define MODE_TYPE_ALL (0xFFFFFFFF) #define MODE_DEFAULT (1 << 0) #define MODE_MEMVERBOSE (1 << 1) +#define MODE_TYPE (1 << 2) /* Display specific type with avg/max */ /* PSI statistics structure */ struct psi_stats { @@ -108,20 +116,28 @@ struct task_info { char command[TASK_COMM_LEN]; unsigned long long cpu_count; unsigned long long cpu_delay_total; + unsigned long long cpu_delay_max; unsigned long long blkio_count; unsigned long long blkio_delay_total; + unsigned long long blkio_delay_max; unsigned long long swapin_count; unsigned long long swapin_delay_total; + unsigned long long swapin_delay_max; unsigned long long freepages_count; unsigned long long freepages_delay_total; + unsigned long long freepages_delay_max; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; + unsigned long long thrashing_delay_max; unsigned long long compact_count; unsigned long long compact_delay_total; + unsigned long long compact_delay_max; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; + unsigned long long wpcopy_delay_max; unsigned long long irq_count; unsigned long long irq_delay_total; + unsigned long long irq_delay_max; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -141,6 +157,7 @@ struct field_desc { const char *cmd_char; /* Interactive command */ unsigned long total_offset; /* Offset of total delay in task_info */ unsigned long count_offset; /* Offset of count in task_info */ + unsigned long max_offset; /* Offset of max delay in task_info */ size_t supported_modes; /* Supported display modes */ }; @@ -153,6 +170,7 @@ struct config { int monitor_pid; /* Monitor specific PID */ char *container_path; /* Path to container cgroup */ const struct field_desc *sort_field; /* Current sort field */ + const struct field_desc *type_field; /* Type field for -t option */ size_t display_mode; /* Current display mode */ }; @@ -164,15 +182,15 @@ static int task_count; static int running = 1; static struct container_stats container_stats; static const struct field_desc sort_fields[] = { - SORT_FIELD(cpu, c, MODE_DEFAULT), - SORT_FIELD(blkio, i, MODE_DEFAULT), - SORT_FIELD(irq, q, MODE_DEFAULT), - SORT_FIELD(mem, m, MODE_DEFAULT | MODE_MEMVERBOSE), - SORT_FIELD(swapin, s, MODE_MEMVERBOSE), - SORT_FIELD(freepages, r, MODE_MEMVERBOSE), - SORT_FIELD(thrashing, t, MODE_MEMVERBOSE), - SORT_FIELD(compact, p, MODE_MEMVERBOSE), - SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE), + SORT_FIELD(cpu, c, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(blkio, i, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(irq, q, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD_NO_MAX(mem, m, MODE_DEFAULT | MODE_MEMVERBOSE), + SORT_FIELD(swapin, s, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(freepages, r, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(thrashing, t, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(compact, p, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE | MODE_TYPE), END_FIELD }; static int sort_selected; @@ -265,6 +283,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" + " -t, --type=FIELD Display only specified delay type with avg/max\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -283,6 +302,7 @@ static void parse_args(int argc, char **argv) {"processes", required_argument, 0, 'P'}, {"sort", required_argument, 0, 's'}, {"container", required_argument, 0, 'C'}, + {"type", required_argument, 0, 't'}, {"memverbose", no_argument, 0, 'M'}, {0, 0, 0, 0} }; @@ -292,6 +312,7 @@ static void parse_args(int argc, char **argv) cfg.iterations = 0; cfg.max_processes = 20; cfg.sort_field = &sort_fields[0]; /* Default sorted by CPU delay */ + cfg.type_field = NULL; /* No type field by default */ cfg.output_one_time = 0; cfg.monitor_pid = 0; /* 0 means monitor all PIDs */ cfg.container_path = NULL; @@ -300,7 +321,7 @@ static void parse_args(int argc, char **argv) while (1) { int option_index = 0; - c = getopt_long(argc, argv, "hd:n:p:oP:C:s:M", long_options, &option_index); + c = getopt_long(argc, argv, "hd:n:p:oP:C:s:t:M", long_options, &option_index); if (c == -1) break; @@ -363,9 +384,32 @@ static void parse_args(int argc, char **argv) cfg.sort_field = field; break; + case 't': + if (strlen(optarg) == 0) { + fprintf(stderr, "Error: empty type field\n"); + exit(1); + } + + field = get_field_by_name(optarg); + /* Show available fields if invalid option provided */ + if (!field || !(field->supported_modes & MODE_TYPE)) { + fprintf(stderr, "Error: invalid type field '%s'\n", optarg); + display_available_fields(MODE_TYPE); + exit(1); + } + + cfg.type_field = field; + cfg.display_mode = MODE_TYPE; + break; case 'M': cfg.display_mode = MODE_MEMVERBOSE; - cfg.sort_field = get_field_by_name("mem"); + /* Find first field supporting MODE_MEMVERBOSE for sorting */ + for (field = sort_fields; field->name != NULL; field++) { + if (field->supported_modes & MODE_MEMVERBOSE) { + cfg.sort_field = field; + break; + } + } break; default: fprintf(stderr, "Try 'delaytop --help' for more information.\n"); @@ -690,7 +734,12 @@ static void fetch_and_fill_task_info(int pid, const char *comm) nested_len = NLA_PAYLOAD(na->nla_len); while (nested_len > 0) { if (nested->nla_type == TASKSTATS_TYPE_STATS) { - memcpy(&stats, NLA_DATA(nested), sizeof(stats)); + size_t payload_len = NLA_PAYLOAD(nested->nla_len); + + memset(&stats, 0, sizeof(stats)); + if (payload_len > sizeof(stats)) + payload_len = sizeof(stats); + memcpy(&stats, NLA_DATA(nested), payload_len); if (task_count < MAX_TASKS) { tasks[task_count].pid = pid; tasks[task_count].tgid = pid; @@ -699,20 +748,28 @@ static void fetch_and_fill_task_info(int pid, const char *comm) tasks[task_count].command[TASK_COMM_LEN - 1] = '\0'; SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); + SET_TASK_STAT(task_count, cpu_delay_max); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); + SET_TASK_STAT(task_count, blkio_delay_max); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); + SET_TASK_STAT(task_count, swapin_delay_max); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); + SET_TASK_STAT(task_count, freepages_delay_max); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); + SET_TASK_STAT(task_count, thrashing_delay_max); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); + SET_TASK_STAT(task_count, compact_delay_max); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); + SET_TASK_STAT(task_count, wpcopy_delay_max); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); + SET_TASK_STAT(task_count, irq_delay_max); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -777,14 +834,14 @@ static int compare_tasks(const void *a, const void *b) const struct task_info *t2 = (const struct task_info *)b; unsigned long long total1; unsigned long long total2; - unsigned long count1; - unsigned long count2; + unsigned long long count1; + unsigned long long count2; double avg1, avg2; total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); total2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->total_offset); - count1 = *(unsigned long *)((char *)t1 + cfg.sort_field->count_offset); - count2 = *(unsigned long *)((char *)t2 + cfg.sort_field->count_offset); + count1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->count_offset); + count2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->count_offset); avg1 = average_ms(total1, count1); avg2 = average_ms(total2, count2); @@ -794,6 +851,26 @@ static int compare_tasks(const void *a, const void *b) return 0; } +/* Get delay values for a specific field */ +static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, + double *avg_ms, double *max_ms) +{ + unsigned long long total, count, max; + + if (!field || !field->max_offset) { + *avg_ms = 0; + *max_ms = 0; + return; + } + + total = *(unsigned long long *)((char *)task + field->total_offset); + count = *(unsigned long long *)((char *)task + field->count_offset); + *avg_ms = average_ms(total, count); + + max = *(unsigned long long *)((char *)task + field->max_offset); + *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ +} + /* Sort tasks by selected field */ static void sort_tasks(void) { @@ -847,7 +924,12 @@ static void get_container_stats(void) while (nl_len > 0) { if (na->nla_type == CGROUPSTATS_TYPE_CGROUP_STATS) { /* Get the cgroupstats structure */ - memcpy(&stats, NLA_DATA(na), sizeof(stats)); + size_t payload_len = NLA_PAYLOAD(na->nla_len); + + memset(&stats, 0, sizeof(stats)); + if (payload_len > sizeof(stats)) + payload_len = sizeof(stats); + memcpy(&stats, NLA_DATA(na), payload_len); /* Fill container stats */ container_stats.nr_sleeping = stats.nr_sleeping; @@ -950,29 +1032,44 @@ static void display_results(int psi_ret) suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", cfg.max_processes, get_name_by_field(cfg.sort_field)); - suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); - if (cfg.display_mode == MODE_MEMVERBOSE) { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", - "MEM(ms)", "SWAP(ms)", "RCL(ms)", - "THR(ms)", "CMP(ms)", "WP(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "---------------------\n"); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + /* Display mode for -t option: show only specified type with avg/max */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + "PID", "TGID", "COMMAND", + "AVG(ms)", "MAX(ms)"); + suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); } else { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", - "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "--------------------------\n"); + suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); + if (cfg.display_mode == MODE_MEMVERBOSE) { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", + "MEM(ms)", "SWAP(ms)", "RCL(ms)", + "THR(ms)", "CMP(ms)", "WP(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "---------------------\n"); + } else { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", + "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "--------------------------\n"); + } } count = task_count < cfg.max_processes ? task_count : cfg.max_processes; for (i = 0; i < count; i++) { - suc &= BOOL_FPRINT(out, "%8d %8d %-15s", + suc &= BOOL_FPRINT(out, "%8d %8d %-17s", tasks[i].pid, tasks[i].tgid, tasks[i].command); - if (cfg.display_mode == MODE_MEMVERBOSE) { + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + double avg_ms, max_ms; + + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + + suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", + avg_ms, max_ms); + } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), TASK_AVG(tasks[i], swapin), -- 2.25.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/3] delaytop: add delay max for delaytop 2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin @ 2026-07-02 22:58 ` Andrew Morton 0 siblings, 0 replies; 9+ messages in thread From: Andrew Morton @ 2026-07-02 22:58 UTC (permalink / raw) To: wang.yaxin Cc: fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 On Thu, 2 Jul 2026 20:58:15 +0800 (CST) <wang.yaxin@zte.com.cn> wrote: > From: Wang Yaxin <wang.yaxin@zte.com.cn> > > Previously delaytop only showed average delays. Add delay_max fields to > track the maximum delay value for each delay type (cpu, blkio, irq, swapin, > freepages, thrashing, compact, wpcopy) per task. > > This provides a global view of all tasks' delay spikes, which is essential > for identifying processes that experienced brief but significant latency > events that would be hidden by average-only metrics. > > The -t/--type option displays only the specified delay type with avg/max > values side by side, allowing focused analysis: > delaytop -t cpu # Show only CPU delay with avg/max > delaytop -t wpcopy # Show Copy-on-Write delay with avg/max > > ... > > total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); The code does this pointer operation in many places. Is there a better way? Should all those `unsigned long long' fields in `struct task_info' be in an array, something like that? ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-07-02 22:58 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-05-27 13:55 [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-05-27 13:57 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin 2026-05-27 13:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin 2026-05-27 13:59 ` [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes wang.yaxin 2026-05-27 19:42 ` [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis Andrew Morton 2026-05-28 4:38 ` wang.yaxin 2026-05-30 3:16 [PATCH 0/3 v2] " wang.yaxin 2026-05-30 3:17 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin 2026-07-02 12:57 [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin 2026-07-02 22:58 ` Andrew Morton
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox