[intel] kernel panic, SNA? #140
Comments
|
Is this reproducible? The crashing process is init; it was reaping another process and was attempting to free its vmspace. We hit a panic in witness, which is a bit odd: the mm rwsem is flagged NOWITNESS. I'm guessing that the process' mm had already been freed. |
|
Briefly looking at the code, I think each vm area should take a reference on current->mm in linux_dev_mmap_single() (or perhaps in the cdev object ctor). Right now, the only references are held by threads, but at the point of the crash all of the threads belonging to the process have been destroyed. @hselasky any thoughts? |
|
Hi @markjdb , The current design assumes that the cdev_pager_dtor() is called from on of the threads belonging to the procedure creating the vm_mm . If that is not the case - adding a reference is the right thing to do. I'm not sure where to add the reference. Are we certain that pager_ctor() is called the same amount of times that pager_dtor() is called? --HPS |
|
It is not really triggerable I think. But it seems that it panics when I'm doing something with epiphany (GNOME's webkit based browser). For example I had 3 or so panics yesterday, but no (yet) today. |
|
Hi @kwm81 I think this might be a sleepable lock after non-sleepable issue. Could you do: gdb --HPS |
|
Hi @markjdb I see that the init process uses the system map, which basically means vm_map_lock() does a mtx_lock() / mtx_unlock() sequence. This will of course trigger witness, because this lock is held by vm_map_remove() across the linux cdev pager dtor. --HPS |
|
Hi @kwm81 Some further analysis reveals that you might have a double panic. I.E. there is a NULL pointer first at address 0x6 and then the panic tries to switch video mode and that fails too causing the second panic. Could you also do: gdb --HPS |
|
Hi Hans, I don't think init's use of a system map should matter here - we should be locking the zombie's map, not init's, right? I wonder if we should instead drop the vmspace ref from the process_exit eventhandler rather than from the thread_dtor. The eventhandler is called first, so vmspace_exit() will release the last reference and free the vmspace (and thus the cdev objects). I still don't understand the root cause of the panic though. |
|
I've asked @kwm81 to try to reproduce the issue. So far no luck. Let's see if we can track this down the one way or the other. |
|
I think I can reproduce this now 100%, by filing a comment in phabricator from epiphany. That is until someone adds a comment to the review I think.... Please let me know the gdb foo to run to get more info! Fatal trap 12: page fault while in kernel mode Fatal trap 12: page fault while in kernel mode |
|
To be specific the two panics above happen when I press the submit button while adding a comment with a code comment in a phab.f.o review. |
|
Can you open up the kernel using GDB and type: info line *(linux_cdev_pager_populate+0x140) --HPS |
|
Probably me doing something wrong but I have no idea why this doesn't work anymore. I suspect that it doesn't load the modules? root@crashalot:~ # gdb /boot/kernel/kernel gdb /usr/lib/debug/boot/kernel/kernel.debugReading symbols from /usr/lib/debug/boot/kernel/kernel.debug...done. |
|
You'll need to load /boot/kernel/linuxkpi.ko aswell. |
|
(gdb) add-symbol-file /usr/lib/debug/boot/kernel/linuxkpi.ko.debug 0 |
|
Can you try this patch: |
|
with the patch, submit a comment in phabricator works. And killing epiphany does not panic the box anymore. I will keep testing the patch and report back if there are any more issues, or if a reasonable time has passed, but it looks good sofar. |
Conflicts: sys/compat/linuxkpi/common/include/asm/smp.h sys/compat/linuxkpi/common/include/linux/smp.h sys/compat/linuxkpi/common/src/linux_compat.c Issue: #140
|
Thank you for testing. Patch is now pushed to drm-next. |
So after running stock HEAD for a while I finaly have time to checkout drm-next again. Then I ran into this gem.
FreeBSD crashalot 12.0-CURRENT FreeBSD 12.0-CURRENT #8 bedc15f(drm-next)
This is a Sandybridge Laptop, running a almost stock GNOME 3 desktop. The intel xorg ddx is configured to use SNA. Normal use consisting of gnome-terminal, gedit and epiphany.
Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address = 0x8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ab383f
stack pointer = 0x28:0xfffffe033ffa0310
frame pointer = 0x28:0xfffffe033ffa0390
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1 (init)
trap number = 12
vt_kms_postswitch() at vt_kms_postswitch+0x52/frame 0xfffffe033ff9fe70
vt_window_switch() at vt_window_switch+0xdb/frame 0xfffffe033ff9feb0
vtterm_cngrab() at vtterm_cngrab+0x20/frame 0xfffffe033ff9fed0
cngrab() at cngrab+0x42/frame 0xfffffe033ff9fef0
vpanic() at vpanic+0x10a/frame 0xfffffe033ff9ff70
panic() at panic+0x43/frame 0xfffffe033ff9ffd0
trap_fatal() at trap_fatal+0x322/frame 0xfffffe033ffa0020
trap_pfault() at trap_pfault+0x62/frame 0xfffffe033ffa0080
trap() at trap+0x29e/frame 0xfffffe033ffa0240
calltrap() at calltrap+0x8/frame 0xfffffe033ffa0240
--- trap 0xc, rip = 0xffffffff80ab383f, rsp = 0xfffffe033ffa0310, rbp = 0xfffffe033ffa0390 ---
witness_checkorder() at witness_checkorder+0x6f/frame 0xfffffe033ffa0390
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !mutex_is_locked(&mode_config->mutex) && !drm_modeset_is_locked(&mode_config->connection_mutex) failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/compat/linuxkpi/common/include/drm/drm_crtc.h:1403
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:909
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:909
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:270
init:100002 WARNING !state->acquire_ctx failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_atomic.c:634
init:100002 WARNING !mutex_is_locked(&mode_config->mutex) && !drm_modeset_is_locked(&mode_config->connection_mutex) failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/compat/linuxkpi/common/include/drm/drm_crtc.h:1403
init:100002 WARNING !mutex_is_locked(&mode_config->mutex) && !drm_modeset_is_locked(&mode_config->connection_mutex) failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/compat/linuxkpi/common/include/drm/drm_crtc.h:1403
WARN_ON(!mutex_is_locked(&dev->struct_mutex))pipe A vblank wait timed out
WARN_ON(!mutex_is_locked(&fbc->lock))WARN_ON(!mutex_is_locked(&fbc->lock))WARN_ON(!mutex_is_locked(&obj->base.dev->struct_mutex))WARN_ON(!mutex_is_locked(&obj->base.dev->struct_mutex))
init:100002 WARNING !mutex_is_locked(&obj->dev->struct_mutex) failed at /home/kwm/sources/freebsd/drm-freebsd-base-graphics/sys/dev/drm/drm_gem.c:854
panic: page fault
cpuid = 6
time = 1492852418
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe033ff9fef0
vpanic() at vpanic+0x19c/frame 0xfffffe033ff9ff70
panic() at panic+0x43/frame 0xfffffe033ff9ffd0
trap_fatal() at trap_fatal+0x322/frame 0xfffffe033ffa0020
trap_pfault() at trap_pfault+0x62/frame 0xfffffe033ffa0080
trap() at trap+0x29e/frame 0xfffffe033ffa0240
calltrap() at calltrap+0x8/frame 0xfffffe033ffa0240
--- trap 0xc, rip = 0xffffffff80ab383f, rsp = 0xfffffe033ffa0310, rbp = 0xfffffe033ffa0390 ---
witness_checkorder() at witness_checkorder+0x6f/frame 0xfffffe033ffa0390
_sx_xlock() at _sx_xlock+0x5e/frame 0xfffffe033ffa03d0
linux_cdev_pager_dtor() at linux_cdev_pager_dtor+0xc2/frame 0xfffffe033ffa03f0
dev_pager_dealloc() at dev_pager_dealloc+0x33/frame 0xfffffe033ffa0410
vm_object_terminate() at vm_object_terminate+0x24e/frame 0xfffffe033ffa0450
vm_object_deallocate() at vm_object_deallocate+0x2d9/frame 0xfffffe033ffa04c0
vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 0xfffffe033ffa04f0
vm_map_remove() at vm_map_remove+0xc8/frame 0xfffffe033ffa0520
vmspace_free() at vmspace_free+0x51/frame 0xfffffe033ffa0540
linux_free_current() at linux_free_current+0x5e/frame 0xfffffe033ffa0560
proc_dtor() at proc_dtor+0xf0/frame 0xfffffe033ffa05b0
uma_zfree_arg() at uma_zfree_arg+0x80/frame 0xfffffe033ffa0600
proc_reap() at proc_reap+0x437/frame 0xfffffe033ffa0640
proc_to_reap() at proc_to_reap+0x36c/frame 0xfffffe033ffa0690
kern_wait6() at kern_wait6+0x2cd/frame 0xfffffe033ffa0740
sys_wait4() at sys_wait4+0x78/frame 0xfffffe033ffa0930
amd64_syscall() at amd64_syscall+0x57a/frame 0xfffffe033ffa0ab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe033ffa0ab0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41e57a, rsp = 0x7fffffffe818, rbp = 0x7fffffffe880 ---
Uptime: 14h53m55s
Dumping 2697 out of 8068 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
The text was updated successfully, but these errors were encountered: