baseline stack overrun in Twitch livestream frames

[NapalmSauce]

(Would have tenderapp been a better place for this?)

That didn't happen before but twitch frames bomb the browser. Example URL with bogus channel but crashes(for me at the very least):
https://embed.twitch.tv/?autoplay=true&channel=Whatever&height=480

If I have baseline enabled (with or without ion) the above makes TFx go down. This can be a nuisance since you might not expect this to be embed on some sites.

From crashreporter, running TFxDebug-fpr23:

Exception Type:  EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0xf1a19ff0
Crashed Thread:  35

Thread 35 crashed with PPC Thread State 32:
  srr0: 0x0c254fc8  srr1: 0x0000f030   dar: 0xf1a19ff0 dsisr: 0x42000000
    r0: 0x0c015614    r1: 0xf1a1a040    r2: 0x062ad800    r3: 0x23cbb000
    r4: 0x35111ba8    r5: 0x35104678    r6: 0x35104408    r7: 0x00000000
    r8: 0x35111ba8    r9: 0x23cbb000   r10: 0x3609dec8   r11: 0xf1a1a120
   r12: 0x0c0155b4   r13: 0xf1a67b30   r14: 0x37f04dc0   r15: 0xffffff88
   r16: 0xf1a1a2ac   r17: 0xf1b10608   r18: 0x0be10a64   r19: 0x3501d044
   r20: 0x0000009c   r21: 0xf1b109c0   r22: 0x00000002   r23: 0xf1b105d8
   r24: 0x00000000   r25: 0x00000000   r26: 0x35200000   r27: 0x35100000
   r28: 0x23cbb000   r29: 0x3501d010   r30: 0x35111ba8   r31: 0x0c0155bc
    cr: 0x24424222   xer: 0x00000003    lr: 0x0c015614   ctr: 0x0c0155b4
vrsave: 0x00000000

(huge) backtrace for thread 35:

0   XUL                0x0c254fc8 js::CurrentThreadCanAccessRuntime(JSRuntime*) + 12
1   XUL                0x0c015610 js::jit::AssertValidObjectPtr(JSContext*, JSObject*) + 92
2   ???                0x24f1c4f4 0 + 619824372
3   XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
4   XUL                0x0be12fa8 js::jit::EnterBaselineMethod(JSContext*, js::RunState&) + 472
5   XUL                0x0c221f9c js::RunScript(JSContext*, js::RunState&) + 488
6   XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
7   XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
8   XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
9   ???                0x26eb78f4 0 + 652966132
10  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
11  XUL                0x0be12fa8 js::jit::EnterBaselineMethod(JSContext*, js::RunState&) + 472
12  XUL                0x0c21e850 __ZL9InterpretP9JSContextRN2js8RunStateE + 49328
13  XUL                0x0c221f10 js::RunScript(JSContext*, js::RunState&) + 348
14  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
15  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
16  XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
17  ???                0x26eb78f4 0 + 652966132
18  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
19  XUL                0x0be12fa8 js::jit::EnterBaselineMethod(JSContext*, js::RunState&) + 472
20  XUL                0x0c221f9c js::RunScript(JSContext*, js::RunState&) + 488
21  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
22  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
23  XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
24  ???                0x26eb78f4 0 + 652966132
25  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
26  XUL                0x0be12fa8 js::jit::EnterBaselineMethod(JSContext*, js::RunState&) + 472
27  XUL                0x0c221f9c js::RunScript(JSContext*, js::RunState&) + 488
28  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
29  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
30  XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
31  ???                0x26eb78f4 0 + 652966132
32  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
33  XUL                0x0be12fa8 js::jit::EnterBaselineMethod(JSContext*, js::RunState&) + 472
34  XUL                0x0c21e850 __ZL9InterpretP9JSContextRN2js8RunStateE + 49328
35  XUL                0x0c221f10 js::RunScript(JSContext*, js::RunState&) + 348
36  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
37  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
38  XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
39  ???                0x26eb78f4 0 + 652966132
40  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
41  XUL                0x0be28224 js::jit::EnterBaselineAtBranch(JSContext*, js::InterpreterFrame*, unsigned char*) + 804
42  XUL                0x0c21ee60 __ZL9InterpretP9JSContextRN2js8RunStateE + 50880
43  XUL                0x0c221f10 js::RunScript(JSContext*, js::RunState&) + 348
44  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
45  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
46  XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
47  ???                0x26eb78f4 0 + 652966132
48  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
49  XUL                0x0be28224 js::jit::EnterBaselineAtBranch(JSContext*, js::InterpreterFrame*, unsigned char*) + 804
50  XUL                0x0c21ee60 __ZL9InterpretP9JSContextRN2js8RunStateE + 50880
51  XUL                0x0c221f10 js::RunScript(JSContext*, js::RunState&) + 348
52  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
53  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
54  XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
55  ???                0x26eb78f4 0 + 652966132
56  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
57  XUL                0x0be12fa8 js::jit::EnterBaselineMethod(JSContext*, js::RunState&) + 472
58  XUL                0x0c21e850 __ZL9InterpretP9JSContextRN2js8RunStateE + 49328
59  XUL                0x0c221f10 js::RunScript(JSContext*, js::RunState&) + 348
60  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
61  XUL                0x0c0c3b8c js::fun_apply(JSContext*, unsigned int, JS::Value*) + 820
62  XUL                0x0c229534 js::CallJSNative(JSContext*, bool (*)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) + 264
63  XUL                0x0c222234 js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 400
64  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
65  XUL                0x0be3325c __ZN2js3jitL14DoCallFallbackEP9JSContextPNS0_13BaselineFrameEPNS0_15ICCall_FallbackEjPN2JS5ValueENS7_13MutableHandleIS8_EE + 1128
66  ???                0x26eb78f4 0 + 652966132
67  XUL                0x0be10a60 __ZL13EnterBaselineP9JSContextRN2js3jit12EnterJitDataE + 392
68  XUL                0x0be12fa8 js::jit::EnterBaselineMethod(JSContext*, js::RunState&) + 472
69  XUL                0x0c221f9c js::RunScript(JSContext*, js::RunState&) + 488
70  XUL                0x0c22233c js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) + 664
71  XUL                0x0c223090 js::Invoke(JSContext*, JS::Value const&, JS::Value const&, unsigned int, JS::Value const*, JS::MutableHandle<JS::Value>) + 640
72  XUL                0x0c03d06c JS::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) + 220
73  XUL                0x09f383cc mozilla::dom::EventHandlerNonNull::Call(JSContext*, JS::Handle<JS::Value>, mozilla::dom::Event&, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&) + 496
74  XUL                0x0a201110 mozilla::JSEventHandler::HandleEvent(nsIDOMEvent*) + 1588
75  XUL                0x0a201930 mozilla::EventListenerManager::HandleEventSubType(mozilla::EventListenerManager::Listener*, nsIDOMEvent*, mozilla::dom::EventTarget*) + 328
76  XUL                0x0a201c34 mozilla::EventListenerManager::HandleEventInternal(nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent**, mozilla::dom::EventTarget*, nsEventStatus*) + 692
77  XUL                0x0a1e56c0 mozilla::EventTargetChainItem::HandleEvent(mozilla::EventChainPostVisitor&, mozilla::ELMCreationDetector&) + 476
78  XUL                0x0a1cc9c4 mozilla::EventTargetChainItem::HandleEventTargetChain(nsTArray<mozilla::EventTargetChainItem>&, mozilla::EventChainPostVisitor&, mozilla::EventDispatchingCallback*, mozilla::ELMCreationDetector&) + 752
79  XUL                0x0a1da374 mozilla::EventDispatcher::Dispatch(nsISupports*, nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent*, nsEventStatus*, mozilla::EventDispatchingCallback*, nsTArray<mozilla::dom::EventTarget*>*) + 3552
80  XUL                0x0a1da6d0 mozilla::EventDispatcher::DispatchDOMEvent(nsISupports*, mozilla::WidgetEvent*, nsIDOMEvent*, nsPresContext*, nsEventStatus*) + 352
81  XUL                0x0a97e850 __ZN12_GLOBAL__N_120MessageEventRunnable16DispatchDOMEventEP9JSContextPN7mozilla3dom7workers13WorkerPrivateEPNS3_20DOMEventTargetHelperEb.isra.1193 + 1164
82  XUL                0x0a948a9c mozilla::dom::workers::WorkerRunnable::Run() + 1096
83  XUL                0x08718c64 nsThread::ProcessNextEvent(bool, bool*) + 844
84  XUL                0x0875c480 NS_ProcessNextEvent(nsIThread*, bool) + 60
85  XUL                0x0a97098c mozilla::dom::workers::WorkerPrivate::DoRunLoop(JSContext*) + 1248
86  XUL                0x0a904630 (anonymous namespace)::WorkerThreadPrimaryRunnable::Run() + 1620
87  XUL                0x08718c64 nsThread::ProcessNextEvent(bool, bool*) + 844
88  XUL                0x0875c480 NS_ProcessNextEvent(nsIThread*, bool) + 60
89  XUL                0x08b69c8c mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) + 360
90  XUL                0x08b3173c MessageLoop::RunInternal() + 148
91  XUL                0x08b317cc MessageLoop::Run() + 64
92  XUL                0x087138b8 nsThread::ThreadFunc(void*) + 396
93  libnss3.dylib      0x07ac88f0 _pt_root + 228
94  libSystem.B.dylib  0x0026ff70 _pthread_start + 316

looking at coredump:

(bad access is 0xf1a19ff0, r1 is 0xf1a1a040 )

(Guard page?)

Load command 1869
      cmd LC_SEGMENT
  cmdsize 56
  segname 
 -----  vmaddr 0xf1a19000
 -----  vmsize 0x00001000
  fileoff 2241069056
 filesize 4096
  maxprot 0x00000007
 initprot 0x00000000
   nsects 0
    flags 0xff000000

(thread's stack segment?)

Load command 1870
      cmd LC_SEGMENT
  cmdsize 56
  segname 
 -----  vmaddr 0xf1a1a000
 -----  vmsize 0x00101000
  fileoff 2241073152
 filesize 1052672
  maxprot 0x00000007
 initprot 0x00000003
   nsects 0
    flags 0xff000000

The backtrace makes it look like it's just stack exhaustion from excessive recursion.
(re-checked with gdb768 to see if things are consistent; they are)

[classilla]

You've pretty much hit the nail on the head -- stack exhaustion is exactly what's going on here.

Unfortunately, we already have a full GB dedicated to the CPU stack, which is already half the available 32-bit addressing space. Disabling the JIT by default doesn't seem like a winning strategy, so since this doesn't work anyway, one way around it might be to unconditionally blacklist one of the Twitch components so it doesn't load.

[NapalmSauce]

What mechanism would be the most appropriate for this?

With more verification today, Twitch.tv has a frontpage livestream and same result happens there, so that nearly entirely prevents twitch from being browsed to.

The crashing script(s) seem to reside in static.twitchcdn.net, and blocking that CDN simply prevents twitch domains from loading.
Even in case someone wants to browse it by any means, tenfourfox's adblock being preffed of by default doesn't make it an ideal option for blacklisting.

It's not obvious how this should be handled. The Intel people and/or others tracking the repo changesets likely will not want this, so smells like#ifdef __ppc__

I can put up a PR and do some testing for next beta(if it's okay with you); just let me know where the blacklist entry should be.

[classilla]

Sure, feel free to take a whack at it. If possible, if you could get a PR up by the end of this week that will give me enough time to pull it in. I think we should do it like #469 but in a block before it and gate it with a different pref which should not exist (tenfourfox.allow_troublesome_js sounds good). If the pref exists and is true, then the scripts are not blocked. The code can otherwise be copy-pasted.

I bet there will be other hosts in this category eventually, so let's have a systematic solution for it.

[NapalmSauce]

(Sorry about the above, I should stop force-pushing my tree)

So I'm using a browser built off NapalmSauce@cf9bf7d and it's working as intended.

I've put a logging function and a logging pref as well. The string matching macro name is BLOC (vs BLOK) to make it easy to tell the blacklist apart from the adblock, should it grow larger.

I think it's ready for you to review, so I'll put up my PR in not long.

[classilla]

Mitigated by #609 but leaving open in case I figure out what's really going on.

[NapalmSauce]

First tag to crash is 45.5.0b1 (45.4.0 survives). #453 had a common regression line. I lack any background in this context to tell if it has to do with hybrid typed-array endianness, but *-leopard-webkit doesn't seem to crash on twitch. Probably not that good a comparison; I can just reverse the JIT portion of 312362 in changesets-20160923 (see comment 4) and let you know what happens. Won't cost me much, and should document better what might be going on.

[NapalmSauce]

Backing that portion of 312362 doesn't change anything about the crash, so maybe the scripts are accessing floats as ints or something else sketchy..

[classilla]

No, I don't think it's an endian bug, because it would show up in the interpreter too. I think either there is a JIT-specific bug, or (more likely) the interpreter just correctly terminates when the stack exhausts and the JIT doesn't.

[classilla]

I'm homing in on CodeGenerator::visitCheckOverRecursed and I suspect it is not adding a recursion check because it thinks we're using less stack than we are (we have famously huge turds). I wonder if the heuristic it's using is not properly adjusted. However, I'm also running out of time for the beta, so I think we'll ship this and hold this over.

[classilla]

Could also be VMFunctions::CheckOverRecursedWithExtra or CheckOverRecursed, though, since you mention this happens even with just Baseline and the Code Generator is Ion-specific.

[classilla]

We'd need some prints in there to see a) if the limit is at the right address and b) that we actually are correctly extracting the stack pointer from the BaselineFrame.

[NapalmSauce]

45.4 and below only had the bug wallpapered since there's no recursion abort in the console. In > 45.5b1 baseline disabled there's a recursion error for script (warning 1.7MB static.twitchcdn.net/assets/worker.min-a76eeba0bf85a2f41428.js). There's emscripten-related names in there, so 45.4 simply halts the script before getting to the crash.

I'm doing a debug build to start investigating the recursion limits.