Kernel error due to "hv_vmbus" .

Question

question

SaiAshrith-6536 asked May 9, '22 SaiAshrith-6536 commented May 24, '22

Kernel error due to "hv_vmbus" .

I am using custom preempt rt kernel 5.10-rt . Whenever i boot the vm in azure i am getting the below error in dmesg logs.

BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:969
[ 757.843701] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/0
[ 757.843703] Preemption disabled at:
[ 757.843703] [<ffffffff82048fde>] schedule_preempt_disabled+0x1e/0x20
[ 757.843712] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.10.100-rt62-skelios #1
[ 757.843715] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 757.843716] Call Trace:
[ 757.843719] <IRQ>
[ 757.843723] dump_stack+0x6b/0x83
[ 757.843727] __might_sleep.cold+0xd6/0xea
[ 757.843731] ? native_safe_halt+0xe/0x10
[ 757.843734] rt_spin_lock+0x1c/0x50
[ 757.843738] vmbus_isr+0x11a/0x330 [hv_vmbus]
[ 757.843750] sysvec_hyperv_callback+0x4a/0x90
[ 757.843755] asm_call_irq_on_stack+0x12/0x20
[ 757.843760] </IRQ>
[ 757.843761] sysvec_hyperv_callback+0xa8/0xe0
[ 757.843764] asm_sysvec_hyperv_callback+0x12/0x20
[ 757.843768] RIP: 0010:native_safe_halt+0xe/0x10
[ 757.843771] Code: 00 00 65 48 8b 04 25 40 bc 01 00 f0 80 60 02 df c3 0f ae f0 0f ae 38 0f ae f0 eb a5 e9 07 00 00 00 0f 00 2d 16 bf 5b 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 06 bf 5b 00 f4 c3 cc cc 0f 1f 44 00
[ 757.843773] RSP: 0018:ffffffff82e03ee8 EFLAGS: 00000206
[ 757.843775] RAX: ffffffff8204c4a0 RBX: ffffffff82e19940 RCX: 0000000000000000
[ 757.843776] RDX: 0000000000000001 RSI: ffffffff828ba1cb RDI: ffffffff82892e8d
[ 757.843778] RBP: 0000000000000000 R08: ffff96a7bbc2af58 R09: 0000000000000014
[ 757.843779] R10: 0000000000000800 R11: 0000000000000000 R12: 00000000000000ab
[ 757.843780] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 757.843782] ? sched_text_end+0x3/0x3
[ 757.843786] ? rcu_dynticks_eqs_enter+0xd/0x30
[ 757.843788] default_idle+0xa/0x10
[ 757.843790] default_idle_call+0x39/0x152
[ 757.843793] do_idle+0x8b/0x100
[ 757.843796] cpu_startup_entry+0x19/0x20
[ 757.843798] start_kernel+0x54e/0x56f
[ 757.843804] secondary_startup_64_no_verify+0xb0/0xbb
[ 759.850810] BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:969
[ 759.850814] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/0
[ 759.850816] Preemption disabled at:
[ 759.850817] [<ffffffff82048fde>] schedule_preempt_disabled+0x1e/0x20
[ 759.850826] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.10.100-rt62-skelios #1
[ 759.850828] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
[ 759.850829] Call Trace:
[ 759.850833] <IRQ>
[ 759.850836] dump_stack+0x6b/0x83
[ 759.850840] __might_sleep.cold+0xd6/0xea
[ 759.850844] ? native_safe_halt+0xe/0x10
[ 759.850848] rt_spin_lock+0x1c/0x50
[ 759.850852] vmbus_isr+0x11a/0x330 [hv_vmbus]
[ 759.850864] __sysvec_hyperv_callback+0x4a/0x90
[ 759.850868] asm_call_irq_on_stack+0x12/0x20
[ 759.850873] </IRQ>
[ 759.850873] sysvec_hyperv_callback+0xa8/0xe0
[ 759.850876] asm_sysvec_hyperv_callback+0x12/0x20
[ 759.850880] RIP: 0010:native_safe_halt+0xe/0x10
[ 759.850883] Code: 00 00 65 48 8b 04 25 40 bc 01 00 f0 80 60 02 df c3 0f ae f0 0f ae 38 0f ae f0 eb a5 e9 07 00 00 00 0f 00 2d 16 bf 5b 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 06 bf 5b 00 f4 c3 cc cc 0f 1f 44 00
[ 759.850885] RSP: 0018:ffffffff82e03ee8 EFLAGS: 00000202
[ 759.850888] RAX: ffffffff8204c4a0 RBX: ffffffff82e19940 RCX: 0000000000000000
[ 759.850889] RDX: 0000000000000001 RSI: ffffffff828ba1cb RDI: ffffffff82892e8d
[ 759.850890] RBP: 0000000000000000 R08: ffff96a7bbc2af58 R09: 000000000000002d
[ 759.850891] R10: 0000000000001400 R11: 0000000000000000 R12: 00000000000000ab
[ 759.850893] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 759.850894] ? __sched_text_end+0x3/0x3
[ 759.850898] ? rcu_dynticks_eqs_enter+0xd/0x30
[ 759.850901] default_idle+0xa/0x10
[ 759.850903] default_idle_call+0x39/0x152
[ 759.850905] do_idle+0x8b/0x100
[ 759.850908] cpu_startup_entry+0x19/0x20
[ 759.850910] start_kernel+0x54e/0x56f
[ 759.850916] secondary_startup_64_no_verify+0xb0/0xbb

Are there any specific kernel configs to be enabled so that we do not get the above error.

azure-virtual-machines-images

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-05-09T21:23:59.83Z

deherman-MSFT answered May 9, '22 SaiAshrith-6536 commented May 24, '22

@SaiAshrith-6536

I understand you are getting the above error when booting your custom Linux VM in Azure. It sounds like you may need to rebuild the initrd so that at least the hv_vmbus and hv_storvsc kernel modules are available on the initial ramdisk. Steps on how to perform this can be found on the preparation page. Please check to see if this works for you.

Hope this helps! If you are still facing issues, please let us know.

Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SaiAshrith-6536 · May 24 at 05:32 AM

Hi @deherman-MSFT

Thanks for your comment,

I have already included the modules (hv_vmbus and hv_storvsc) in the initrd image, i think the error will be different if the modules are not present.
Actually the issue is happening only in RT Kernel, and may be it is due to the driver (hv_vmbus) which is not handled with RT patches are applied.

0 ·

question