gc2-es: Fix dimm-util err_inj related error#2742
Open
Joseph-Shih-ww wants to merge 1 commit into
Open
Conversation
[Task Description] * Related to GC20T5T7-131、 GC20T5T7-230 * After injecting a DIMM PMIC error that causes the system to shut down, the following expected behavior was not observed: 1. host power off 2. err_list display the correct error type [Motivation] * Expect the behavior of gc2 to align with that of yv35, which has the same architecture [Design] * Refer to yv35 and implement a method that reads via I3C whenever DC OFF is detected. [Test Plan] 1. Verify that there are no errors in the current DIMM $ dimm-util server --pmic --err_list 2. DIMM PMIC error injection $ dimm-util server --pmic --dimm 0 --err_inj <error> 3. Check error $ dimm-util server --pmic --err_list $ log-util all --print | tail -20 4. Check power status $ power-util server status 5. Recovery $ power-util server 12V-cycle [Test Log] ``` [SWAout_OV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj SWAout_OV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: SWAout_OV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 07 23:28:50 log-util: User cleared all logs 0 all 2026-06-07 23:28:50 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-07 23:28:50 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (001A41) PVCCIN_CPU0 status: 0x1a41 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (007C00) PVCCIN_CPU0 status: 0x7c00 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-06-07 23:34:55 gpiod FRU: 1, Server is powered off 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (007F01) PVCCIN_CPU0 status: 0x7f01 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: VR_FAULT (0xB1), Event Data: (008010) PVCCIN_CPU0 status: 0x8010 Assertion 1 server 2026-06-07 23:34:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:34:55, Sensor: PMIC_ERROR (0xB4), Event Data: (000000) PMIC_ERROR DIMM A2 SWAOUT_OV Assertion 1 server 2026-06-07 23:35:00 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:35:00, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-07 23:35:09 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:35:09, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [SWCout_OV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj SWCout_OV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: SWCout_OV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 07 23:48:08 log-util: User cleared all logs 0 all 2026-06-07 23:48:09 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-07 23:48:09 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-07 23:48:44 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System PXE boot fail, Fail Type: IPv6 fail, Error Code: 0x10 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (001A41) PVCCIN_CPU0 status: 0x1a41 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (007C00) PVCCIN_CPU0 status: 0x7c00 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (007F01) PVCCIN_CPU0 status: 0x7f01 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: VR_FAULT (0xB1), Event Data: (008010) PVCCIN_CPU0 status: 0x8010 Assertion 1 server 2026-06-07 23:52:49 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:49, Sensor: PMIC_ERROR (0xB4), Event Data: (000200) PMIC_ERROR DIMM A2 SWCOUT_OV Assertion 1 server 2026-06-07 23:52:49 gpiod FRU: 1, Server is powered off 1 server 2026-06-07 23:52:54 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:52:54, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-07 23:53:03 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-07 23:53:03, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [SWDout_OV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj SWDout_OV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: SWDout_OV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 08 00:00:21 log-util: User cleared all logs 0 all 2026-06-08 00:00:21 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-08 00:00:21 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-08 00:00:28 ipmid SEL Entry: FRU: 1, Record: Facebook Unified SEL (0xFB), GeneralInfo: POST(0x28), POST Failure Event: System HTTP boot fail, Fail Type: IPv6 fail, Error Code: 0x63 1 server 2026-06-08 00:00:56 gpiod FRU: 1, Server is powered off 1 server 2026-06-08 00:00:56 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:00:56, Sensor: PMIC_ERROR (0xB4), Event Data: (000300) PMIC_ERROR DIMM A2 SWDOUT_OV Assertion 1 server 2026-06-08 00:01:01 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:01:01, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-08 00:01:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:01:10, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [VinB_OV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj VinB_OV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: VinB_OV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 08 01:39:59 log-util: User cleared all logs 0 all 2026-06-08 01:40:00 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-08 01:40:00 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (001A41) PVCCIN_CPU0 status: 0x1a41 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (007C00) PVCCIN_CPU0 status: 0x7c00 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (007F01) PVCCIN_CPU0 status: 0x7f01 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: VR_FAULT (0xB1), Event Data: (008010) PVCCIN_CPU0 status: 0x8010 Assertion 1 server 2026-06-08 01:40:30 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:30, Sensor: PMIC_ERROR (0xB4), Event Data: (000400) PMIC_ERROR DIMM A2 VIN_BULK_OV Assertion 1 server 2026-06-08 01:40:31 gpiod FRU: 1, Server is powered off 1 server 2026-06-08 01:40:35 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:35, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-08 01:40:44 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:40:44, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [SWAout_UV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj SWAout_UV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: SWAout_UV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 08 00:36:48 log-util: User cleared all logs 0 all 2026-06-08 00:36:51 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-08 00:36:51 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (001A41) PVCCIN_CPU0 status: 0x1a41 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (007C00) PVCCIN_CPU0 status: 0x7c00 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (007F01) PVCCIN_CPU0 status: 0x7f01 Assertion 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: VR_FAULT (0xB1), Event Data: (008010) PVCCIN_CPU0 status: 0x8010 Assertion 1 server 2026-06-08 00:37:55 gpiod FRU: 1, Server is powered off 1 server 2026-06-08 00:37:55 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:37:55, Sensor: PMIC_ERROR (0xB4), Event Data: (000600) PMIC_ERROR DIMM A2 SWAOUT_UV Assertion 1 server 2026-06-08 00:38:00 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:38:00, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-08 00:38:09 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:38:09, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [SWCout_UV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj SWCout_UV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: SWCout_UV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 08 00:56:15 log-util: User cleared all logs 0 all 2026-06-08 00:56:18 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-08 00:56:18 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (001A41) PVCCIN_CPU0 status: 0x1a41 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (007C00) PVCCIN_CPU0 status: 0x7c00 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (007F01) PVCCIN_CPU0 status: 0x7f01 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: VR_FAULT (0xB1), Event Data: (008010) PVCCIN_CPU0 status: 0x8010 Assertion 1 server 2026-06-08 00:58:46 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:46, Sensor: PMIC_ERROR (0xB4), Event Data: (000800) PMIC_ERROR DIMM A2 SWCOUT_UV Assertion 1 server 2026-06-08 00:58:46 gpiod FRU: 1, Server is powered off 1 server 2026-06-08 00:58:51 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:58:51, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-08 00:59:00 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 00:59:00, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [SWDout_UV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj SWDout_UV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: SWDout_UV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 08 01:19:35 log-util: User cleared all logs 0 all 2026-06-08 01:19:36 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-08 01:19:36 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (001A41) PVCCIN_CPU0 status: 0x1a41 Assertion 1 server 2026-06-08 01:20:12 gpiod FRU: 1, Server is powered off 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (007C00) PVCCIN_CPU0 status: 0x7c00 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (007F01) PVCCIN_CPU0 status: 0x7f01 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: VR_FAULT (0xB1), Event Data: (008010) PVCCIN_CPU0 status: 0x8010 Assertion 1 server 2026-06-08 01:20:12 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:12, Sensor: PMIC_ERROR (0xB4), Event Data: (000900) PMIC_ERROR DIMM A2 SWDOUT_UV Assertion 1 server 2026-06-08 01:20:17 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:17, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-08 01:20:26 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:20:26, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [VinB_UV] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj VinB_UV Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: VinB_UV DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 08 02:02:18 log-util: User cleared all logs 0 all 2026-06-08 02:02:18 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-08 02:02:18 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (001A41) PVCCIN_CPU0 status: 0x1a41 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (007841) PVCCIN_CPU0 status: 0x7841 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (007A00) PVCCIN_CPU0 status: 0x7a00 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (007B00) PVCCIN_CPU0 status: 0x7b00 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (007C00) PVCCIN_CPU0 status: 0x7c00 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (007D00) PVCCIN_CPU0 status: 0x7d00 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (007E00) PVCCIN_CPU0 status: 0x7e00 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (007F01) PVCCIN_CPU0 status: 0x7f01 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: VR_FAULT (0xB1), Event Data: (008010) PVCCIN_CPU0 status: 0x8010 Assertion 1 server 2026-06-08 02:04:10 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:10, Sensor: PMIC_ERROR (0xB4), Event Data: (000A00) PMIC_ERROR DIMM A2 VIN_BULK_UV Assertion 1 server 2026-06-08 02:04:10 gpiod FRU: 1, Server is powered off 1 server 2026-06-08 02:04:15 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:15, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-08 02:04:24 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 02:04:24, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1... [critical_temp_shutdown] root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: No Error DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# dimm-util server --pmic --dimm 0 --err_inj critical_temp_shutdown Error inject successfully on DIMM A2 root@bmc-oob:/tmp# dimm-util server --pmic --err_list DIMM A2 PMIC Error: critical_temp_shutdown DIMM A3 PMIC Error: No DIMM DIMM A6 PMIC Error: No DIMM DIMM A7 PMIC Error: No DIMM root@bmc-oob:/tmp# log-util all --print | tail -20 2026 Jun 08 01:46:03 log-util: User cleared all logs 0 all 2026-06-08 01:46:03 healthd ASSERT: Verified boot failure (4,41) 0 all 2026-06-08 01:46:03 healthd Verified boot failure reason: The intermediate keys were not verified using the ROM keys 1 server 2026-06-08 01:46:22 gpiod FRU: 1, Server is powered off 1 server 2026-06-08 01:46:22 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:46:22, Sensor: PMIC_ERROR (0xB4), Event Data: (001000) PMIC_ERROR DIMM A2 CRITICAL_TEMP_SHUTDOWN Assertion 1 server 2026-06-08 01:46:27 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:46:27, Sensor: ME_POWER_STATE (0x16), Event Data: (000000) RUNNING Assertion 1 server 2026-06-08 01:46:36 ipmid SEL Entry: FRU: 1, Record: Standard (0x02), Time: 2026-06-08 01:46:36, Sensor: SYSTEM_STATUS (0x10), Event Data: (0AFFFF) SYS_VRWATCHDOG Assertion root@bmc-oob:/tmp# power-util server status Power status for fru 1 : OFF root@bmc-oob:~# power-util server 12V-cycle 12V Power cycling fru 1...
meta-cla
Bot
added
the
CLA Signed
Contributor
|
This pull request has been imported. If you are a Meta employee, you can view this in D110744409. (Because this pull request was imported automatically, there will not be any future comments.) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Task Description]
[Motivation]
[Design]
[Test Plan]
1 . Verify that there are no errors in the current DIMM
2 . DIMM PMIC error injection
3 . Check error
4 . Check power status
5 . Recovery
[Test Log]
[SWAout_OV]
[SWCout_OV]
[SWDout_OV]
[VinB_OV]
[SWAout_UV]
[SWCout_UV]
[SWDout_UV]
[VinB_UV]
[critical_temp_shutdown]