1. 介绍

本文主要介绍crash工具的使用以及常用的命令。crash工具，常用来分析内核的coredump以及应用的coredump，功能非常强大。

crash工具官方介绍

使用crash分析内核crash情况，需要准备以下内容：

内核crash时生成的coredump文件或者raw data

运行内核对应的符号表vmlinux

crash 工具

2. crash基本使用

2.1 解释命令

使用crash工具解析内核crash产生的raw data，命令如下：

crash cluster-vmlinux ramdump@0x57400000

解析成功后，会进入crash工具cmdline模式，如下图所示：

使用crash工具解析内核crash产生的coredump文件，命令如下：

crash vmlinux vmcore

解析成功后，也会进入crash工具的cmdline模式

2.2 crash工具基本输出

KERNEL: cluster-vmlinux [TAINTED]DUMPFILES: /var/tmp/ramdump_elf_Tp0C6d [temporary ELF header]ramdumpCPUS: 6DATE: Mon Apr 18 11:33:58 CST 2022 //panic时间UPTIME: 01:28:40 //panic时，已运行时间LOAD AVERAGE: 1.31, 1.32, 1.29TASKS: 134 //panic时，进程总数NODENAME: x9h_ms //硬件信息RELEASE: 4.14.61 //内核版本信息VERSION: #1 SMP PREEMPT Fri Apr 15 14:25:47 UTC 2022MACHINE: aarch64 (unknown Mhz)MEMORY: 544 MB //Linux系统内存数PANIC: "sysrq: SysRq : Trigger a crash" //panic原因PID: 2496 //panic进程号COMMAND: "sh" //panic进程名词TASK: ffff800023c5b800 [THREAD_INFO: ffff800023c5b800] //panic进程task和thread_info信息CPU: 5 //系统cpu核数STATE: TASK_RUNNING (SYSRQ) //panic时任务状态

2.2.1 crash常用命令

help
该命令是查看crash工具的帮助信息，譬如支持的命令、具体某个命令的使用方法等。

crash> help* extend log rd taskalias files mach repeat timerascii foreach mod runq treebpf fuser mount search unionbt gdb net set vmbtop help p sig vtopdev ipcs ps struct waitqdis irq pte swap whatiseval kmem ptob sym wrexit list ptov sys qcrash version: 8.0.0++ gdb version: 10.2For help on any command above, enter "help <command>".For help on input options, enter "help input".For help on output options, enter "help output".//查看某个命令使用方法crash> help psNAMEps - display process status informationSYNOPSISps [-k|-u|-G|-y policy] [-s] [-p|-c|-t|-[l|m][-C cpu]|-a|-g|-r|-S|-A][pid | task | command] ...DESCRIPTIONThis command displays process status for selected, or all, processesin the system. If no arguments are entered, the process data isis displayed for all processes. Specific processes may be selectedby using the following identifier formats:pid a process PID.task a hexadecimal task_struct pointer.command a command name. If a command name is made up of letters thatare all numerical values, precede the name string with a "\".If the command string is enclosed within "'" characters, thenthe encompassed string must be a POSIX extended regular expressionthat will be used to match task names.The process list may be further restricted by the following options:-k restrict the output to only kernel threads.-u restrict the output to only user tasks.-G display only the thread group leader in a thread group.-y policy restrict the output to tasks having a specified scheduling policyexpressed by its integer value or by its (case-insensitive) name;multiple policies may be entered in a comma-separated list:0 or NORMAL1 or FIFO2 or RR3 or BATCH4 or ISO5 or IDLE6 or DEADLINEThe process identifier types may be mixed. For each task, the followingitems are displayed:1. the process PID.2. the parent process PID.3. the CPU number that the task ran on last.4. the task_struct address or the kernel stack pointer of the process.(see -s option below)5. the task state (RU, IN, UN, ZO, ST, TR, DE, SW, WA, PA, ID, NE).6. the percentage of physical memory being used by this task.7. the virtual address size of this task in kilobytes.8. the resident set size of this task in kilobytes.9. the command name.The default output shows the task_struct address of each process under acolumn titled "TASK". This can be changed to show the kernel stackpointer under a column titled "KSTACKP".-s replace the TASK column with the KSTACKP column.On SMP machines, the active task on each CPU will be highlighted by anangle bracket (">") preceding its information. If the crash variable"offline" is set to "hide", the active task on an offline CPU willbe highlighted by a "-" preceding its information.Alternatively, information regarding parent-child relationships,per-task time usage data, argument/environment data, thread groups,or resource limits may be displayed:-p display the parental hierarchy of selected, or all, tasks.-c display the children of selected, or all, tasks.-t display the task run time, start time, and cumulative userand system times.-l display the task's last-run timestamp value, using either thetask_struct's last_run value, the task_struct's timestamp valueor the task_struct's sched_entity last_arrival value, whicheverapplies, of selected, or all, tasks; the list is sorted with themost recently-run task (with the largest timestamp) shown first,followed by the task's current state.-m similar to -l, but the timestamp value is translated into days,hours, minutes, seconds, and milliseconds since the task waslast run on a cpu.-C cpus only usable with the -l or -m options, dump the timestamp datain per-cpu blocks, where the cpu[s] can be specified as "1,3,5","1-3", "1,3,5-7,10", "all", or "a" (shortcut for "all").-a display the command line arguments and environment strings ofselected, or all, user-mode tasks.-g display tasks by thread group, of selected, or all, tasks.-r display resource limits (rlimits) of selected, or all, tasks.-S display a summary consisting of the number of tasks in a task state.-A display only the active task on each cpu.EXAMPLESShow the process status of all current tasks:crash> psPID PPID CPU TASK ST %MEM VSZ RSS COMM> 0 0 3 c024c000 RU 0.0 0 0 [swapper]> 0 0 0 c0dce000 RU 0.0 0 0 [swapper]0 0 1 c0fa8000 RU 0.0 0 0 [swapper]> 0 0 2 c009a000 RU 0.0 0 0 [swapper]1 0 1 c0098000 IN 0.0 1096 476 init2 1 1 c0090000 IN 0.0 0 0 [kflushd]3 1 1 c000e000 IN 0.0 0 0 [kpiod]4 1 3 c000c000 IN 0.0 0 0 [kswapd]5 1 1 c0008000 IN 0.0 0 0 [mdrecoveryd]253 1 2 fbc4c000 IN 0.0 1088 376 portmap

bt命令
该命令是查看进程栈信息的，可以通过pid/cpu等选项指定相关信息

crash> btPID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"#0 [ffff000013573800] __crash_kexec at ffff00000817cee8#1 [ffff000013573890] (null) at 108142390#2 [ffff000013573920] psci_sys_reset at ffff000008965ae0#3 [ffff000013573940] machine_restart at ffff000008085b94#4 [ffff000013573960] emergency_restart at ffff0000080fddc4#5 [ffff000013573970] panic at ffff0000080d8c04#6 [ffff000013573a50] die at ffff00000808ae10#7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8#8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc#9 [ffff000013573b30] do_translation_fault at ffff0000080a0390#10 [ffff000013573b40] do_mem_abort at ffff00000808130c#11 [ffff000013573d20] el1_ia at ffff000008083050PC: ffff0000086095b0 [sysrq_handle_crash+32]LR: ffff00000860959c [sysrq_handle_crash+12]SP: ffff000013573d30 PSTATE: 60400145X29: ffff000013573d30 X28: ffff800023c5b800 X27: ffff000008cc1000X26: 0000000000000040 X25: 0000000000000124 X24: 0000000000000000X23: 0000000000000004 X22: ffff000009217000 X21: ffff000009217f10X20: 0000000000000063 X19: ffff00000919e000 X18: 0000000000000010X17: 000000000049b4d8 X16: ffff000008277b80 X15: ffffffffffffffffX14: ffff0000892fc7a7 X13: ffff0000092fc7b5 X12: ffff00000919e000X11: ffff00000917ae78 X10: ffff000008434f78 X9: 00000000ffffffd0X8: 0000000000000015 X7: 6767697254203a20 X6: 00000000000004a3X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000X2: ffff80002731efb8 X1: 0000000000000000 X0: 0000000000000001#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c#16 [ffff000013573dc0] __vfs_write at ffff00000827760c#17 [ffff000013573e40] vfs_write at ffff000008277900#18 [ffff000013573e80] sys_write at ffff000008277bc4#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abcPC: 0000ffff9e5aaeac LR: 0000000000408cc4 SP: 0000ffffeea390d0X29: 0000ffffeea390d0 X28: 0000000000000000 X27: 0000000000000000X26: 00000000071a3cc5 X25: 000000000049b000 X24: 00000000071a6372X23: 0000000000000000 X22: 0000000000000001 X21: 00000000071a6370X20: 0000000000000002 X19: 0000000000000001 X18: 00000000000002c8X17: 000000000049b4d8 X16: 0000ffff9e5aae80 X15: 0000ffff9e4ebde0X14: 0000ffff9e4f92c8 X13: 000000000000270f X12: 0101010101010101X11: 0000000000000000 X10: 0101010101010101 X9: fffffffffffffff0X8: 0000000000000040 X7: 7f7f7f7f7f7f7f7f X6: 0080000080808080X5: 0000000000000000 X4: 00000000071a0063 X3: 0000ffff9e63f190X2: 0000000000000002 X1: 00000000071a6370 X0: 0000000000000001ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 80000000

set
该命令获取crash的进程信息

crash> setPID: 2496COMMAND: "sh"TASK: ffff800023c5b800 [THREAD_INFO: ffff800023c5b800]CPU: 5STATE: TASK_RUNNING (SYSRQ)

sym
该命令查看符号和符号对应的地址信息，可以根据符号给出对应的地址，也可以根据给出的地址给出对应的符号

crash> sym proc_reg_writeffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224crash> sym ffff0000082ea1d0ffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224crash> sym __log_bufffff0000092ff020 (b) __log_bufcrash> sym ffff0000092ff020ffff0000092ff020 (b) __log_buf

ps
查看系统的进程信息，类似linux的ps

crash> psPID PPID CPU TASK ST %MEM VSZ RSS COMM> 0 0 0 ffff000009186300 RU 0.0 0 0 [swapper/0]> 0 0 1 ffff800025ba9c00 RU 0.0 0 0 [swapper/1]> 0 0 2 ffff800025baaa00 RU 0.0 0 0 [swapper/2]> 0 0 3 ffff800025bab800 RU 0.0 0 0 [swapper/3]> 0 0 4 ffff800025bac600 RU 0.0 0 0 [swapper/4]0 0 5 ffff800025bad400 RU 0.0 0 0 [swapper/5]1 0 2 ffff800025b70000 IN 0.1 1824 1276 init2 0 2 ffff800025b70e00 IN 0.0 0 0 [kthreadd]4 2 0 ffff800025b72a00 ID 0.0 0 0 [kworker/0:0H]6 2 0 ffff800025b74600 ID 0.0 0 0 [mm_percpu_wq]7 2 0 ffff800025b75400 IN 0.0 0 0 [ksoftirqd/0]8 2 5 ffff800025b76200 ID 0.0 0 0 [rcu_preempt]9 2 1 ffff800025b77000 ID 0.0 0 0 [rcu_sched]10 2 0 ffff800025ba8000 ID 0.0 0 0 [rcu_bh]11 2 0 ffff800025ba8e00 IN 0.0 0 0 [migration/0]12 2 0 ffff800025bae200 IN 0.0 0 0 [cpuhp/0]13 2 1 ffff800025baf000 IN 0.0 0 0 [cpuhp/1]14 2 1 ffff800025be8000 IN 0.0 0 0 [migration/1]...

vtop
根据一个虚拟地址转换成对应的物理地址

crash> vtop ffff800025065400VIRTUAL PHYSICALffff800025065400 65065400PAGE DIRECTORY: ffff00000938b000PGD: ffff00000938b800 => 673f8803PUD: ffff8000273f8000 => 673f7803PMD: ffff8000273f7940 => f8000065000f11PAGE: 65000000 (2MB)PTE PHYSICAL FLAGSf8000065000f11 65000000 (VALID|SHARED|AF|NG|PXN|UXN|DIRTY)PAGE PHYSICAL MAPPING INDEX CNT FLAGSffff7e0000941940 65065000 dead000000000400 0 0 fffc00000000000

ptov
根据一个物理地址，转存成对应的内核态虚拟地址

crash> ptov 65065400VIRTUAL PHYSICALffff800025065400 65065400

struct
查看结构体信息，如成员偏移，某个结构体变量内存中的值等

crash> struct thread_info -ostruct thread_info {[0] unsigned long flags;[8] mm_segment_t addr_limit;[16] int preempt_count;}SIZE: 24crash> struct thread_info -o -xstruct thread_info {[0x0] unsigned long flags;[0x8] mm_segment_t addr_limit;[0x10] int preempt_count;}SIZE: 0x18crash> struct thread_info 0xffff000009186300struct thread_info {flags = 32,addr_limit = 281474976710655,preempt_count = 65537}crash> task_struct.thread_info 0xffff000009186300 -pthread_info = {flags = 32,addr_limit = 281474976710655,preempt_count = 65537},

dis
反汇编，根据给出的PC地址，反编译出汇编源码，一般结合bt提供的栈信息和寄存器信息和反编译的汇编源码信息分析。

crash> btPID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"#0 [ffff000013573800] __crash_kexec at ffff00000817cee8#1 [ffff000013573890] (null) at 108142390#2 [ffff000013573920] psci_sys_reset at ffff000008965ae0#3 [ffff000013573940] machine_restart at ffff000008085b94#4 [ffff000013573960] emergency_restart at ffff0000080fddc4#5 [ffff000013573970] panic at ffff0000080d8c04#6 [ffff000013573a50] die at ffff00000808ae10#7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8#8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc#9 [ffff000013573b30] do_translation_fault at ffff0000080a0390#10 [ffff000013573b40] do_mem_abort at ffff00000808130c#11 [ffff000013573d20] el1_ia at ffff000008083050PC: ffff0000086095b0 [sysrq_handle_crash+32]LR: ffff00000860959c [sysrq_handle_crash+12]SP: ffff000013573d30 PSTATE: 60400145X29: ffff000013573d30 X28: ffff800023c5b800 X27: ffff000008cc1000X26: 0000000000000040 X25: 0000000000000124 X24: 0000000000000000X23: 0000000000000004 X22: ffff000009217000 X21: ffff000009217f10X20: 0000000000000063 X19: ffff00000919e000 X18: 0000000000000010X17: 000000000049b4d8 X16: ffff000008277b80 X15: ffffffffffffffffX14: ffff0000892fc7a7 X13: ffff0000092fc7b5 X12: ffff00000919e000X11: ffff00000917ae78 X10: ffff000008434f78 X9: 00000000ffffffd0X8: 0000000000000015 X7: 6767697254203a20 X6: 00000000000004a3X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000X2: ffff80002731efb8 X1: 0000000000000000 X0: 0000000000000001#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c#16 [ffff000013573dc0] __vfs_write at ffff00000827760c#17 [ffff000013573e40] vfs_write at ffff000008277900#18 [ffff000013573e80] sys_write at ffff000008277bc4#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abcPC: 0000ffff9e5aaeac LR: 0000000000408cc4 SP: 0000ffffeea390d0X29: 0000ffffeea390d0 X28: 0000000000000000 X27: 0000000000000000X26: 00000000071a3cc5 X25: 000000000049b000 X24: 00000000071a6372X23: 0000000000000000 X22: 0000000000000001 X21: 00000000071a6370X20: 0000000000000002 X19: 0000000000000001 X18: 00000000000002c8X17: 000000000049b4d8 X16: 0000ffff9e5aae80 X15: 0000ffff9e4ebde0X14: 0000ffff9e4f92c8 X13: 000000000000270f X12: 0101010101010101X11: 0000000000000000 X10: 0101010101010101 X9: fffffffffffffff0X8: 0000000000000040 X7: 7f7f7f7f7f7f7f7f X6: 0080000080808080X5: 0000000000000000 X4: 00000000071a0063 X3: 0000ffff9e63f190X2: 0000000000000002 X1: 00000000071a6370 X0: 0000000000000001ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 80000000//查看栈数据crash> bt -f 2PID: 2 TASK: ffff800179d30000 CPU: 1 COMMAND: "kthreadd"#0 [ffff00000a56bd90] __switch_to at ffff000008086690ffff00000a56bd90: ffff00000a56bdb0 ffff000008ea5fe8 ffff00000a56bda0: ffff80017feae280 ffff800179d6e580 #1 [ffff00000a56bdb0] __schedule at ffff000008ea5fe4ffff00000a56bdb0: ffff00000a56be40 ffff000008ea6708 ffff00000a56bdc0: ffff800179d30000 ffff000009a2bda0 ffff00000a56bdd0: ffff000009b8df60 ffff8000f4276600 ffff00000a56bde0: 0000000000000000 ffff000008108b68 ffff00000a56bdf0: 0000000000000001 ffff000008108000 ffff00000a56be00: 0000000000000000 0000000000000000 ffff00000a56be10: ffff000008109f54 ffff00000a56be60 ffff00000a56be20: ffff000008109e80 80838e611e66f600 ffff00000a56be30: ffff000000000004 80838e611e66f600 #2 [ffff00000a56be40] schedule at ffff000008ea6704ffff00000a56be40: ffff00000a56be60 ffff000008109f54 ffff00000a56be50: ffff8000f4276628 ffff000008109e80 #3 [ffff00000a56be60] kthreadd at ffff000008109f50//查看异常栈crash> bt -ePID: 1803 TASK: ffff8001794fab80 CPU: 5 COMMAND: "pvr_defer_free"KERNEL-MODE EXCEPTION FRAME AT: ffff00000e7dbbc0PC: ffff00000876d444 [DeviceMemSet+140]LR: ffff0000087609c0 [_ZeroPageArray+120]SP: ffff00000e7dbd00 PSTATE: 20c00145X29: ffff00000e7dbd00 X28: ffff80015bc9fb00 X27: 00000000000186a0X26: 0000000000000000 X25: 00e8000000000f0f X24: 0000000000000080X23: ffff80012d0db000 X22: ffff00001fb7c000 X21: 0000000000000000X20: 0000000000080000 X19: ffff00001fbaf000 X18: 0000ffffac000bccX17: 0000ffff97c22ec8 X16: ffff00000818ded0 X15: 0000ffffac000bc8X14: 0140000000000000 X13: ffff00001fbfc000 X12: 0000000000000000X11: 0000000000000000 X10: 0000000000000040 X9: 0040000000000041X8: 0040000000000001 X7: 0000000000000001 X6: 000000017fffd7e8X5: ffff8001398feb98 X4: ffff8001398feb98 X3: ffff00001fbfbfffX2: 0000000000000000 X1: 0000000000000000 X0: ffff00001fbfc000

irq
查看系统的中断信息

crash> irqIRQ IRQ_DESC/_DATA IRQACTION NAME0 (unused) (unused)1 ffff800025807400 (unused)2 ffff800025807800 (unused)3 ffff800025807a00 (unused)4 ffff800025807c00 ffff80002588df00 "arch_timer"5 ffff800025807e00 (unused)6 ffff800024c7e400 (unused)7 ffff800024c7e600 ffff800024fcc100 "ttyS0"8 ffff800024c7e800 (unused)9 ffff800024c7ea00 ffff800024b10680 "30b00000.i2c"10 ffff800024c7ec00 ffff800024efc200 "30b10000.i2c"11 ffff800024c7ee00 ffff800024c72f00 "30b30000.i2c"...

files
查看进程打开的文件

crash> filesPID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"ROOT: / CWD: /home/rootFD FILE DENTRY INODE TYPE PATH0 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS01 ffff800024de9500 ffff8000239b29c0 ffff80002399e508 REG /proc/sysrq-trigger2 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS010 ffff8000236a3b00 ffff8000245ff300 ffff800025095668 CHR /dev/tty11 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS0crash> foreach files -R /dev/console //结合foreach命令搜索打开/dev/console文件的进程PID: 2331 TASK: ffff800024a1b800 CPU: 2 COMMAND: "messagecenter"ROOT: / CWD: /FD FILE DENTRY INODE TYPE PATH1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/consolePID: 2460 TASK: ffff800024f8aa00 CPU: 5 COMMAND: "adbd"ROOT: / CWD: /FD FILE DENTRY INODE TYPE PATH1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/consolePID: 2462 TASK: ffff80002421f000 CPU: 2 COMMAND: "usb ffs open"ROOT: / CWD: /FD FILE DENTRY INODE TYPE PATH1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/consolePID: 2463 TASK: ffff80002421aa00 CPU: 5 COMMAND: "server socket"ROOT: / CWD: /FD FILE DENTRY INODE TYPE PATH1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/consolePID: 2464 TASK: ffff800024a1aa00 CPU: 5 COMMAND: "->transport"ROOT: / CWD: /FD FILE DENTRY INODE TYPE PATH1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/consolePID: 2465 TASK: ffff800024a1c600 CPU: 5 COMMAND: "<-transport"ROOT: / CWD: /FD FILE DENTRY INODE TYPE PATH1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/consolePID: 2478 TASK: ffff800024a1d400 CPU: 1 COMMAND: "start_getty"ROOT: / CWD: /FD FILE DENTRY INODE TYPE PATH0 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console1 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console2 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console

dev
查看系统注册的设备信息

crash> devCHRDEV NAME CDEV OPERATIONS1 mem ffff8000253f5b00 memory_fops2 pty ffff800024a6ed80 tty_fops3 ttyp ffff8000249e3280 tty_fops4 /dev/vc/0 ffff00000934b7a8 console_fops4 tty ffff80002487e180 tty_fops4 ttyS ffff800024f02e80 tty_fops5 /dev/tty ffff00000934a2d0 tty_fops5 /dev/console ffff00000934a338 console_fops5 /dev/ptmx ffff00000934a4b0 ptmx_fops...BLKDEV NAME GENDISK OPERATIONS259 blkext (none)7 loop ffff800024e85000 lo_fops8 sd (none)9 md ffff800023db8800 md_fops31 mtdblock (none)...

mount
查看系统挂载的文件系统

crash> mountMOUNT SUPERBLK TYPE DEVNAME DIRNAMEffff800025b4c000 ffff800025812000 rootfs rootfs /ffff800023d0c000 ffff800023db9000 ext4 /dev/root /ffff800023d0c1c0 ffff8000250b8000 devtmpfs devtmpfs /devffff800023e0c000 ffff800025815000 proc proc /procffff800023e0c1c0 ffff800023e60000 sysfs sysfs /sysffff800023e0c380 ffff800025817000 debugfs debugfs /sys/kernel/debugffff800023e0c540 ffff80002403f800 configfs configfs /sys/kernel/configffff800023e0c700 ffff800023e60800 tmpfs tmpfs /runffff800023e0c8c0 ffff800023e61000 tmpfs tmpfs /var/volatileffff8000250a8540 ffff8000234e4000 devpts devpts /dev/ptsffff8000250a8700 ffff800023e61000 tmpfs tmpfs /var/libffff800023e0ca80 ffff80002403f800 configfs none /configffff800023e0cc40 ffff800023e62000 functionfs adb /dev/usb-ffs/adbffff800023d0c380 ffff800024e19800 tracefs tracefs /sys/kernel/debug/tracingffff8000250a88c0 ffff8000234e6800 ext4 /dev/block/by-name/userdata /data

kmem
查看内核态下内存的使用情况，如slab信息、vmalloc信息等

crash> kmem -iPAGES TOTAL PERCENTAGETOTAL MEM 120690 471.4 MB ----FREE 76395 298.4 MB 63% of TOTAL MEMUSED 44295 173 MB 36% of TOTAL MEMSHARED 7202 28.1 MB 5% of TOTAL MEMBUFFERS 649 2.5 MB 0% of TOTAL MEMCACHED 10890 42.5 MB 9% of TOTAL MEMSLAB 7869 30.7 MB 6% of TOTAL MEMTOTAL HUGE 0 0 ----HUGE FREE 0 0 0% of TOTAL HUGETOTAL SWAP 0 0 ----SWAP USED 0 0 0% of TOTAL SWAPSWAP FREE 0 0 0% of TOTAL SWAPCOMMIT LIMIT 60345 235.7 MB ----COMMITTED 43807 171.1 MB 72% of TOTAL LIMITcrash> kmem -s //查看kmalloc信息CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAMEffff800024ada180 136 3052 3060 102 4k ext4_groupinfo_1kffff800023c96000 144 3 28 1 4k ext4_groupinfo_4kffff800024164480 1408 2 46 2 32k UDPv6ffff800024164300 232 0 0 0 4k tw_sock_TCPv6ffff800024164180 304 0 0 0 8k request_sock_TCPv6ffff800024164000 2112 5 75 5 32k TCPv6ffff8000241c7e00 224 233 882 49 4k nf_conntrack_expectffff8000241c7c80 256 0 0 0 8k nf_conntrackffff8000241c7b00 312 0 0 0 8k ashmem_area_cacheffff8000241c7980 3312 0 0 0 32k kcopyd_jobffff8000241c7800 120 0 0 0 4k dm_rq_target_ioffff8000241c7680 72 0 0 0 4k isp1760_qtdffff8000241c7500 112 13 216 6 4k cfq_io_cqffff8000241c7380 240 14 102 6 4k cfq_queueffff8000241c7200 216 0 0 0 4k bsg_cmdffff8000241c7080 896 1 17 1 16k mqueue_inode_cacheffff8000241c6f00 656 0 0 0 16k v9fs_inode_cache...crash> kmem -S //查看所有slab对象信息CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAMEffff800024ada180 136 3052 3060 102 4k ext4_groupinfo_1kCPU 0 KMEM_CACHE_CPU:ffff7dffbfecbfb0CPU 0 SLAB:(empty)CPU 0 PARTIAL:(empty)CPU 1 KMEM_CACHE_CPU:ffff7dffbfee4fb0CPU 1 SLAB:SLAB MEMORY NODE TOTAL ALLOCATED FREEffff7e0000721f80 ffff80001c87e000 0 30 22 8FREE / [ALLOCATED][ffff80001c87e000][ffff80001c87e088]...crash> kmem -p //查看page信息PAGE PHYSICAL MAPPING INDEX CNT FLAGSffff7e0000000000 40000000 0 0 0 0ffff7e0000000040 40001000 0 0 0 0ffff7e0000000080 40002000 0 0 0 0ffff7e00000000c0 40003000 0 0 0 0ffff7e0000000100 40004000 0 0 0 0ffff7e0000000140 40005000 0 0 0 0...

foreach
遍历系统上task的信息，如查看系统上处于UNINTERRUPT状态的task的栈信息

crash> foreach UN btPID: 561 TASK: ffff8000e3d33a00 CPU: 0 COMMAND: "kworker/u12:1"#0 [ffff00002038b5f0] __switch_to at ffff000008086690#1 [ffff00002038b610] __schedule at ffff000008ea5fe4#2 [ffff00002038b6a0] schedule at ffff000008ea6704#3 [ffff00002038b6c0] wait_transaction_locked at ffff0000083b5d8c#4 [ffff00002038b730] add_transaction_credits at ffff0000083b602c#5 [ffff00002038b7a0] start_this_handle at ffff0000083b624c#6 [ffff00002038b840] jbd2__journal_start at ffff0000083b6a50#7 [ffff00002038b8a0] __ext4_journal_start_sb at ffff0000083564d8#8 [ffff00002038b8f0] ext4_writepages at ffff0000083760cc#9 [ffff00002038ba40] do_writepages at ffff000008225400#10 [ffff00002038bac0] __writeback_single_inode at ffff0000082ee774#11 [ffff00002038bb10] writeback_sb_inodes at ffff0000082ef180#12 [ffff00002038bc10] __writeback_inodes_wb at ffff0000082ef4b0#13 [ffff00002038bc60] wb_writeback at ffff0000082ef828#14 [ffff00002038bd10] wb_workfn at ffff0000082f0400#15 [ffff00002038bdd0] process_one_work at ffff000008101950#16 [ffff00002038be20] worker_thread at ffff000008101c6c#17 [ffff00002038be70] kthread at ffff000008108c9cPID: 814 TASK: ffff8001793f8000 CPU: 0 COMMAND: "saftey_heart_be"#0 [ffff00000c053c10] __switch_to at ffff000008086690#1 [ffff00000c053c30] __schedule at ffff000008ea5fe4#2 [ffff00000c053cc0] schedule at ffff000008ea6704#3 [ffff00000c053ce0] schedule_timeout at ffff000008eaa428#4 [ffff00000c053d80] msleep at ffff000008178d64#5 [ffff00000c053da0] saftey_hb_thread at ffff000008ad925c#6 [ffff00000c053e70] kthread at ffff000008108c9cPID: 1767 TASK: ffff800020776580 CPU: 0 COMMAND: "AVDemuxer::Read"#0 [ffff000015dc3780] __switch_to at ffff000008086690#1 [ffff000015dc37a0] __schedule at ffff000008ea5fe4#2 [ffff000015dc3830] schedule at ffff000008ea6704#3 [ffff000015dc3850] wait_transaction_locked at ffff0000083b5d8c#4 [ffff000015dc38c0] add_transaction_credits at ffff0000083b602c#5 [ffff000015dc3930] start_this_handle at ffff0000083b624c#6 [ffff000015dc39d0] jbd2__journal_start at ffff0000083b6a50#7 [ffff000015dc3a30] __ext4_journal_start_sb at ffff0000083564d8#8 [ffff000015dc3a80] ext4_da_write_begin at ffff000008378448#9 [ffff000015dc3c10] generic_perform_write at ffff00000821263c#10 [ffff000015dc3ca0] __generic_file_write_iter at ffff000008215884#11 [ffff000015dc3cf0] ext4_file_write_iter at ffff000008362760#12 [ffff000015dc3d80] __vfs_write at ffff0000082b5180#13 [ffff000015dc3e30] vfs_write at ffff0000082b53e0#14 [ffff000015dc3e70] sys_write at ffff0000082b56f8#15 [ffff000015dc3ff0] el0_svc_naked at ffff000008083af4PC: 0000ffffa8eaee60 LR: 0000ffffa8eaee48 SP: 0000ffff93ffcdc0X29: 0000ffff93ffcdc0 X28: 0000000000008000 X27: 0000000002faf080X26: 0000000000000000 X25: 0000000000000000 X24: 0000000000001000X23: 0000ffff84004de0 X22: 0000000000001000 X21: 0000ffff84004de0X20: 0000000000001000 X19: 000000000000001f X18: 0000ffffa8f5ba70X17: 0000ffffa8e50398 X16: 0000ffffa9b00420 X15: 0000074895555910X14: 0000000000000000 X13: 52e6647d9bae90b9 X12: 936c08cc82e6992cX11: 2de7fcffe5393767 X10: 41a928c963510c66 X9: 44377654e95c11b2X8: 0000000000000040 X7: 042d1f3b698dddd1 X6: 0000000000000000X5: 0000ffff93ffede0 X4: 00000000ffffffbb X3: 0000000000000000X2: 0000000000001000 X1: 0000ffff84004de0 X0: 000000000000001fORIG_X0: 000000000000001f SYSCALLNO: 40 PSTATE: 80000000...

p
打印变量或者表达式的值

crash> p jiffiesjiffies = $1 = 4298283231crash> p vm_event_statesPER-CPU DATA TYPE:struct vm_event_state vm_event_states;PER-CPU ADDRESSES:[0]: ffff80017fe91ab8[1]: ffff80017feaaab8[2]: ffff80017fec3ab8[3]: ffff80017fedcab8[4]: ffff80017fef5ab8[5]: ffff80017ff0eab8//查看cpu1对应的变量的值crash> p vm_event_states:1per_cpu(vm_event_states, 1) = $3 = {event = {318552, 739584, 0, 0, 558108, 2889268, 0, 0, 0, 0, 0, 0, 0, 4189877, 38713, 21935, 255, 2454551, 259, 0, 27971, 34455, 0, 63883, 0, 0, 0, 0, 62080, 233, 2, 1, 14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4576, 0, 0, 0, 46844, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 622, 34, 48, 0, 0, 0, 0, 0, 2119, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0}}//查看数组元素crash> p &page_wait_table$4 = (wait_queue_head_t (*)[256]) 0xffff000009a07d80 <page_wait_table>crash> p page_wait_table[4]$5 = {lock = {{rlock = {raw_lock = {owner = 6378,next = 6378}}}},head = {next = 0xffff000009a07de8 <page_wait_table+104>,prev = 0xffff000009a07de8 <page_wait_table+104>}}

list
遍历由指针或者链表连接起来的内容

//查看挂在panic_notifier_list的notifycrash> p &panic_notifier_list $3 = (struct atomic_notifier_head *) 0xffff000009b8a670 <panic_notifier_list>crash> struct atomic_notifier_head struct atomic_notifier_head {spinlock_t lock;struct notifier_block *head;}SIZE: 16crash> struct notifier_blockstruct notifier_block {notifier_fn_t notifier_call;struct notifier_block *next;int priority;}SIZE: 24crash> list atomic_notifier_head.head 0xffff000009b8a670 -s notifier_block.nextffff000009b8a670next = 0xffff000009afaad0 <sdrv_wdt_panic_event_nb>,ffff000009afaad0next = 0xffff000009a3b540 <trace_panic_notifier>,ffff000009a3b540next = 0xffff000009a322a0 <rcu_panic_block>,ffff000009a322a0next = 0xffff000009a3ad50 <panic_block>,ffff000009a3ad50next = 0xffff000009aa5b80 <crashdump_panic_event_nb>,ffff000009aa5b80next = 0xffff000009a1b190 <kernel_offset_notifier>,ffff000009a1b190next = 0xffff000009a1b588 <cpu_hwcaps_notifier>,ffff000009a1b588next = 0xffff000009a1d3b0 <mem_limit_notifier>,ffff000009a1d3b0next = 0xffff000009afd460 <heartbeat_panic_nb>,ffff000009afd460next = 0x0,