Skip to content

test terminates during execution #272

Open
Open
test terminates during execution#272

Description

@puneet336
puneet336
opened on Jun 21, 2026

Hi Team
While trying to run gpu fryer using enroot, the run abruptly stops with "Killed" message,
OS: Ubuntu 22.04.5 LTS
Kernel: 5.15.0-1063-nvidia

Here are the logs:

root@server ~/gpu_fryer# enroot start  gpu_fryer
Detected GPU #0: "NVIDIA H100 80GB HBM3" 
Detected GPU #1: "NVIDIA H100 80GB HBM3"
Detected GPU #2: "NVIDIA H100 80GB HBM3"
Detected GPU #3: "NVIDIA H100 80GB HBM3" 
Detected GPU #4: "NVIDIA H100 80GB HBM3" 
Detected GPU #5: "NVIDIA H100 80GB HBM3"
Detected GPU #6: "NVIDIA H100 80GB HBM3"
Detected GPU #7: "NVIDIA H100 80GB HBM3" 
Using precision(s):  BF16
Creating random matrices
Matrices created
GPU #0: Using 72504 MB
GPU #1: Using 72504 MB
GPU #2: Using 72504 MB
GPU #3: Using 72504 MB
GPU #4: Using 72504 MB
GPU #5: Using 72504 MB
GPU #6: Using 72504 MB
GPU #7: Using 72504 MB
761 (836728 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) | Temperatures: 46°C - 32°C - 32°C - 29°C - 30°C - 31°C - 32°C - 31°C | Throttling: None - None - None - None - None - None - None - None
693 (761961 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) | Temperatures: 49°C - 32°C - 32°C - 29°C - 30°C - 31°C - 32°C - 31°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) | Temperatures: 51°C - 32°C - 32°C - 29°C - 30°C - 31°C - 32°C - 31°C | Throttling: None - None - None - None - None - None - None - None
694 (763061 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 0 (0 Gflops/s) - 785 (863116 Gflops/s) - 0 (0 Gflops/s) | Temperatures: 52°C - 32°C - 32°C - 29°C - 30°C - 31°C - 45°C - 31°C | Throttling: None - None - None - None - None - None - None - None
695 (764160 Gflops/s) - 1161 (1276532 Gflops/s) - 894 (982963 Gflops/s) - 1072 (1178676 Gflops/s) - 931 (1023645 Gflops/s) - 0 (0 Gflops/s) - 701 (770757 Gflops/s) - 1097 (1206164 Gflops/s) | Temperatures: 53°C - 47°C - 49°C - 48°C - 45°C - 31°C - 49°C - 50°C | Throttling: None - None - None - None - None - None - None - None
694 (763061 Gflops/s) - 700 (769658 Gflops/s) - 696 (765260 Gflops/s) - 678 (745468 Gflops/s) - 689 (757563 Gflops/s) - 753 (827932 Gflops/s) - 699 (768558 Gflops/s) - 686 (754264 Gflops/s) | Temperatures: 53°C - 51°C - 52°C - 50°C - 46°C - 46°C - 50°C - 51°C | Throttling: None - None - None - None - None - None - None - None
693 (761961 Gflops/s) - 697 (766359 Gflops/s) - 691 (759762 Gflops/s) - 680 (747667 Gflops/s) - 685 (753165 Gflops/s) - 692 (760862 Gflops/s) - 697 (766359 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 53°C - 52°C - 53°C - 51°C - 47°C - 49°C - 50°C - 52°C | Throttling: None - None - None - None - None - None - None - None
695 (764160 Gflops/s) - 697 (766359 Gflops/s) - 692 (760862 Gflops/s) - 680 (747667 Gflops/s) - 688 (756463 Gflops/s) - 691 (759762 Gflops/s) - 697 (766359 Gflops/s) - 686 (754264 Gflops/s) | Temperatures: 54°C - 54°C - 54°C - 51°C - 47°C - 51°C - 51°C - 53°C | Throttling: None - None - None - None - None - None - None - None
693 (761961 Gflops/s) - 698 (767459 Gflops/s) - 688 (756463 Gflops/s) - 684 (752065 Gflops/s) - 688 (756463 Gflops/s) - 690 (758663 Gflops/s) - 698 (767459 Gflops/s) - 687 (755364 Gflops/s) | Temperatures: 54°C - 55°C - 55°C - 52°C - 48°C - 52°C - 52°C - 54°C | Throttling: None - None - None - None - None - None - None - None
693 (761961 Gflops/s) - 697 (766359 Gflops/s) - 691 (759762 Gflops/s) - 682 (749866 Gflops/s) - 688 (756463 Gflops/s) - 690 (758663 Gflops/s) - 696 (765260 Gflops/s) - 687 (755364 Gflops/s) | Temperatures: 55°C - 55°C - 56°C - 52°C - 49°C - 53°C - 52°C - 55°C | Throttling: None - None - None - None - None - None - None - None
694 (763061 Gflops/s) - 696 (765260 Gflops/s) - 690 (758663 Gflops/s) - 684 (752065 Gflops/s) - 686 (754264 Gflops/s) - 691 (759762 Gflops/s) - 697 (766359 Gflops/s) - 687 (755364 Gflops/s) | Temperatures: 55°C - 56°C - 57°C - 52°C - 49°C - 54°C - 53°C - 56°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 696 (765260 Gflops/s) - 690 (758663 Gflops/s) - 685 (753165 Gflops/s) - 687 (755364 Gflops/s) - 689 (757563 Gflops/s) - 695 (764160 Gflops/s) - 686 (754264 Gflops/s) | Temperatures: 56°C - 57°C - 57°C - 53°C - 50°C - 55°C - 53°C - 57°C | Throttling: None - None - None - None - None - None - None - None
693 (761961 Gflops/s) - 697 (766359 Gflops/s) - 688 (756463 Gflops/s) - 684 (752065 Gflops/s) - 686 (754264 Gflops/s) - 691 (759762 Gflops/s) - 697 (766359 Gflops/s) - 686 (754264 Gflops/s) | Temperatures: 56°C - 58°C - 58°C - 53°C - 50°C - 56°C - 54°C - 57°C | Throttling: None - None - None - None - None - None - None - None
690 (758663 Gflops/s) - 694 (763061 Gflops/s) - 689 (757563 Gflops/s) - 685 (753165 Gflops/s) - 691 (759762 Gflops/s) - 690 (758663 Gflops/s) - 697 (766359 Gflops/s) - 686 (754264 Gflops/s) | Temperatures: 56°C - 58°C - 59°C - 54°C - 50°C - 56°C - 55°C - 57°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 694 (763061 Gflops/s) - 690 (758663 Gflops/s) - 685 (753165 Gflops/s) - 688 (756463 Gflops/s) - 688 (756463 Gflops/s) - 695 (764160 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 56°C - 59°C - 59°C - 54°C - 51°C - 57°C - 55°C - 57°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 695 (764160 Gflops/s) - 689 (757563 Gflops/s) - 686 (754264 Gflops/s) - 689 (757563 Gflops/s) - 687 (755364 Gflops/s) - 696 (765260 Gflops/s) - 686 (754264 Gflops/s) | Temperatures: 57°C - 59°C - 60°C - 54°C - 51°C - 57°C - 55°C - 58°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 695 (764160 Gflops/s) - 689 (757563 Gflops/s) - 685 (753165 Gflops/s) - 688 (756463 Gflops/s) - 688 (756463 Gflops/s) - 696 (765260 Gflops/s) - 685 (753165 Gflops/s) | Temperatures: 57°C - 60°C - 60°C - 55°C - 51°C - 58°C - 56°C - 58°C | Throttling: None - None - None - None - None - None - None - None
690 (758663 Gflops/s) - 695 (764160 Gflops/s) - 688 (756463 Gflops/s) - 683 (750966 Gflops/s) - 687 (755364 Gflops/s) - 687 (755364 Gflops/s) - 693 (761961 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 57°C - 60°C - 61°C - 55°C - 52°C - 58°C - 56°C - 58°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 693 (761961 Gflops/s) - 687 (755364 Gflops/s) - 685 (753165 Gflops/s) - 688 (756463 Gflops/s) - 686 (754264 Gflops/s) - 696 (765260 Gflops/s) - 685 (753165 Gflops/s) | Temperatures: 58°C - 60°C - 61°C - 55°C - 51°C - 58°C - 56°C - 59°C | Throttling: None - None - None - None - None - None - None - None
694 (763061 Gflops/s) - 694 (763061 Gflops/s) - 685 (753165 Gflops/s) - 685 (753165 Gflops/s) - 687 (755364 Gflops/s) - 687 (755364 Gflops/s) - 694 (763061 Gflops/s) - 685 (753165 Gflops/s) | Temperatures: 58°C - 60°C - 60°C - 55°C - 52°C - 59°C - 57°C - 59°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 693 (761961 Gflops/s) - 687 (755364 Gflops/s) - 685 (753165 Gflops/s) - 686 (754264 Gflops/s) - 685 (753165 Gflops/s) - 693 (761961 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 58°C - 61°C - 61°C - 55°C - 52°C - 59°C - 57°C - 59°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 695 (764160 Gflops/s) - 687 (755364 Gflops/s) - 685 (753165 Gflops/s) - 686 (754264 Gflops/s) - 687 (755364 Gflops/s) - 694 (763061 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 58°C - 61°C - 62°C - 56°C - 53°C - 60°C - 57°C - 59°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 694 (763061 Gflops/s) - 687 (755364 Gflops/s) - 685 (753165 Gflops/s) - 686 (754264 Gflops/s) - 686 (754264 Gflops/s) - 695 (764160 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 58°C - 62°C - 62°C - 56°C - 53°C - 60°C - 58°C - 59°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 692 (760862 Gflops/s) - 687 (755364 Gflops/s) - 683 (750966 Gflops/s) - 687 (755364 Gflops/s) - 684 (752065 Gflops/s) - 692 (760862 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 58°C - 62°C - 62°C - 56°C - 53°C - 60°C - 58°C - 59°C | Throttling: None - None - None - None - None - None - None - None
693 (761961 Gflops/s) - 694 (763061 Gflops/s) - 688 (756463 Gflops/s) - 685 (753165 Gflops/s) - 687 (755364 Gflops/s) - 686 (754264 Gflops/s) - 693 (761961 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 59°C - 62°C - 62°C - 56°C - 53°C - 61°C - 58°C - 60°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 693 (761961 Gflops/s) - 689 (757563 Gflops/s) - 685 (753165 Gflops/s) - 688 (756463 Gflops/s) - 685 (753165 Gflops/s) - 694 (763061 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 59°C - 62°C - 63°C - 56°C - 53°C - 61°C - 59°C - 60°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 693 (761961 Gflops/s) - 690 (758663 Gflops/s) - 685 (753165 Gflops/s) - 685 (753165 Gflops/s) - 685 (753165 Gflops/s) - 694 (763061 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 59°C - 63°C - 63°C - 56°C - 54°C - 61°C - 59°C - 60°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 697 (766359 Gflops/s) - 691 (759762 Gflops/s) - 685 (753165 Gflops/s) - 687 (755364 Gflops/s) - 685 (753165 Gflops/s) - 696 (765260 Gflops/s) - 684 (752065 Gflops/s) | Temperatures: 59°C - 63°C - 64°C - 56°C - 54°C - 61°C - 59°C - 60°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 699 (768558 Gflops/s) - 689 (757563 Gflops/s) - 686 (754264 Gflops/s) - 687 (755364 Gflops/s) - 685 (753165 Gflops/s) - 695 (764160 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 59°C - 63°C - 64°C - 57°C - 54°C - 62°C - 60°C - 61°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 700 (769658 Gflops/s) - 692 (760862 Gflops/s) - 685 (753165 Gflops/s) - 687 (755364 Gflops/s) - 686 (754264 Gflops/s) - 694 (763061 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 59°C - 63°C - 64°C - 57°C - 54°C - 62°C - 60°C - 61°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 698 (767459 Gflops/s) - 693 (761961 Gflops/s) - 682 (749866 Gflops/s) - 686 (754264 Gflops/s) - 685 (753165 Gflops/s) - 694 (763061 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 59°C - 63°C - 64°C - 57°C - 54°C - 62°C - 60°C - 61°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 699 (768558 Gflops/s) - 694 (763061 Gflops/s) - 683 (750966 Gflops/s) - 687 (755364 Gflops/s) - 686 (754264 Gflops/s) - 696 (765260 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 59°C - 63°C - 64°C - 57°C - 55°C - 63°C - 60°C - 61°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 700 (769658 Gflops/s) - 694 (763061 Gflops/s) - 683 (750966 Gflops/s) - 686 (754264 Gflops/s) - 689 (757563 Gflops/s) - 696 (765260 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 59°C - 64°C - 65°C - 57°C - 55°C - 63°C - 60°C - 61°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 699 (768558 Gflops/s) - 693 (761961 Gflops/s) - 684 (752065 Gflops/s) - 687 (755364 Gflops/s) - 689 (757563 Gflops/s) - 696 (765260 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 60°C - 64°C - 65°C - 58°C - 54°C - 63°C - 60°C - 61°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 698 (767459 Gflops/s) - 693 (761961 Gflops/s) - 684 (752065 Gflops/s) - 685 (753165 Gflops/s) - 689 (757563 Gflops/s) - 694 (763061 Gflops/s) - 682 (749866 Gflops/s) | Temperatures: 60°C - 64°C - 65°C - 58°C - 55°C - 63°C - 61°C - 61°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 698 (767459 Gflops/s) - 694 (763061 Gflops/s) - 682 (749866 Gflops/s) - 685 (753165 Gflops/s) - 689 (757563 Gflops/s) - 695 (764160 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 60°C - 64°C - 65°C - 58°C - 55°C - 63°C - 61°C - 61°C | Throttling: None - None - None - None - None - None - None - None
692 (760862 Gflops/s) - 699 (768558 Gflops/s) - 693 (761961 Gflops/s) - 683 (750966 Gflops/s) - 687 (755364 Gflops/s) - 690 (758663 Gflops/s) - 695 (764160 Gflops/s) - 683 (750966 Gflops/s) | Temperatures: 60°C - 64°C - 65°C - 58°C - 55°C - 63°C - 61°C - 61°C | Throttling: None - None - None - None - None - None - None - None
691 (759762 Gflops/s) - 699 (768558 Gflops/s) - 694 (763061 Gflops/s) - 683 (750966 Gflops/s) - 686 (754264 Gflops/s) - 689 (757563 Gflops/s) - 694 (763061 Gflops/s) - 682 (749866 Gflops/s) | Temperatures: 60°C - 65°C - 65°C - 58°C - 55°C - 64°C - 61°C - 62°C | Throttling: None - None - None - None - None - None - None - None
Killed

no error stacktrace.
please advice.

UPDATE:

i reattempted using:
enroot start gpu_fryer --use-fp8

and i see:

| Temperatures: 52°C - 57°C - 57°C - 49°C - 48°C - 56°C - 55°C - 53°C | Throttling: None - None - None - None - None - None - None - None
1228 (1350200 Gflops/s) - 1237 (1360095 Gflops/s) - 1224 (1345802 Gflops/s) - 1234 (1356797 Gflops/s) - 1234 (1356797 Gflops/s) - 1219 (1340304 Gflops/s) - 1237 (1360095 Gflops/s) - 1234 (1356797 Gflops/s) | Temperatures: 52°C - 57°C - 57°C - 49°C - 48°C - 56°C - 56°C - 53°C | Throttling: None - None - None - None - None - None - None - None
GPU #0: 1355084 Gflops/s (min: 1348001.26, max: 1358996.37, dev: 1355084.16)
         Temperature: 51.07°C (min: 48.00, max: 52.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
GPU #1: 1360352 Gflops/s (min: 1357896.86, max: 1368891.98, dev: 1360351.58)
         Temperature: 54.37°C (min: 50.00, max: 57.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
GPU #2: 1347618 Gflops/s (min: 1344702.72, max: 1354598.33, dev: 1347617.71)
         Temperature: 54.72°C (min: 50.00, max: 57.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
GPU #3: 1358996 Gflops/s (min: 1355697.84, max: 1365593.44, dev: 1358996.37)
         Temperature: 47.37°C (min: 42.00, max: 49.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
GPU #4: 1358076 Gflops/s (min: 1346901.74, max: 1362294.91, dev: 1358075.85)
         Temperature: 46.35°C (min: 42.00, max: 48.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
GPU #5: 1346262 Gflops/s (min: 1340304.67, max: 1355697.84, dev: 1346262.49)
         Temperature: 53.56°C (min: 47.00, max: 56.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
GPU #6: 1368304 Gflops/s (min: 1355697.84, max: 1645968.91, dev: 1368303.87)
         Temperature: 52.49°C (min: 46.00, max: 56.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
GPU #7: 1362525 Gflops/s (min: 1348001.26, max: 1621779.65, dev: 1362525.04)
         Temperature: 51.00°C (min: 45.00, max: 53.00)
         Throttling HW: false, Thermal SW: false, Thermal HW: false
All GPUs seem healthy
Freeing GPUs...

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions