Electron process error with mapbox-gl

[smit1678]

Current deployment runs into an electron error with mapbox-gl. Unclear if this is related to a version or dependency.

Docker output from the renderer:

debug: renderer received message {"email":"nate@hotosm.org","map":{"lat":57.71096311750432,"lng":11.95305096853906,"zoom":12.399017242352361,"startDate":1500807600000,"endDate":1532343600000,"interval":"months","style":{"background":"light","features":[{"base":{"line-color":{"a":0.7,"b":187,"g":147,"r":42},"line-width":1},"name":"roads","enabled":true,"highlight":{"line-color":{"a":0.3,"b":255,"g":255,"r":255},"line-width":1},"baseEnabled":true,"highlightEnabled":true},{"base":{"line-color":{"a":0.3,"b":109,"g":85,"r":214},"line-width":1},"name":"buildings","enabled":true,"highlight":{"line-color":{"a":0.3,"b":255,"g":255,"r":255},"line-width":1},"baseEnabled":true,"highlightEnabled":true}]}},"format":"mp4","speed":1,"size":"1280x720","dir":"8b815230ec7002d3a3cc90ccfba3b440"}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: 2017-07-23 11:00:00 (0/12 [months])
 {}
info: electron process: error: renderer timeouted, exiting... {}
 {}
debug: electron rendering finsished {}
error: exited with 1 {}

[dakotabenjamin]

To add to this:
Once this error occurs, any future render requests are ignored. They show up in the api logs, but nothing in the renderer logs.

[szymonkaliski]

So I remember seeing this when the docker container with electron was running low on RAM, could you try increasing memory that docker can allocate?

Is this happening on your local machine, or some hosted env?

Also, is there a reproducible way to trigger this?

[szymonkaliski]

Also, I just noticed develop branch, so where is that error happening? In master or develop?

If in develop, then I suspect this commit might be an issue: 227936a

[smit1678]

If in develop, then I suspect this commit might be an issue: 227936a

Yeah I thought we had reverted that. That was a shot in the dark because of the error was being seen both on Elastic Beanstalk and the Digital Ocean box.

But I think the work from @dakotabenjamin last night thought that memory might be an issue because the issue was popping up in other places.

Any recommendations on how we manage memory across containers?

[szymonkaliski]

Any recommendations on how we manage memory across containers?

Not sure what you mean here, sorry. But I'd recommend turning on swap on the boxes you run this on, and monitoring to see if that's actually the problem.

Ideally we could find some kind of reproducible crash, that we can backtrace and fix this issue...

[dakotabenjamin]

We can manage memory with the Dockerrun.aws.json file- I gave each container 256MiB of memory. I will try increasing the memory allocation for the renderer. What do you recommend?

[szymonkaliski]

Oh much more. So what the renderer is doing is: crating virtual X server, running Electron with WebGL inside of it, and software-rendering the window content (mapboxgl map with lines). I'm pretty sure we'd need more for all the textures. I'm afraid this could be trial&error process, but I'd start with 1GB.

[dakotabenjamin]

What other containers do you think will require a bump in memory? I was thinking I should put them all at least to 512MiB.

[szymonkaliski]

Other containers are pretty small, I wouldn't worry about them. If we are working in some kinds of constraints, I think it's better to give as much as we can to the renderer, and leave others small. The node API is super simple, the frontend is basically running static http server in production mode, ngnix, rabbitmq and postgres don't need much. Only the renderer is resource-hungry.

[dakotabenjamin]

We have a t2.large with 8gb, so I set the max memory allocation for the renderer to 2GiB and everything else 512MiB. If that doesn't work I will adjust accordingly.

[szymonkaliski]

👍 let me know how it goes!

[dakotabenjamin]

I bumped it up to 4GB but unfortunately it still fails:

debug: renderer received message {"email":"dakota.benjamin@hotosm.org","map":{"lat":34.391820000000024,"lng":132.45209,"zoom":12.49999999999995,"startDate":1531843200000,"endDate":1532448000000,"interval":"days","style":{"background":"light","features":[{"name":"roads","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":42,"g":147,"b":187,"a":0.7},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}},{"name":"buildings","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":214,"g":85,"b":109,"a":0.3},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}}]}},"format":"gif","speed":1,"size":"1280x720","dir":"2edc4eb019f8968b12c398a89b4e65e8"}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: 2018-07-17 16:00:00 (0/7 [days])
 {}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}

editing this to add on, I am sometimes able to get it to work I've been playing around with format/size (although I'm not finding a pattern yet):

debug: renderer received message {"email":"dakota.benjamin@hotosm.org","map":{"lat":34.391820000000024,"lng":132.45209,"zoom":12.49999999999995,"startDate":1531843200000,"endDate":1532448000000,"interval":"days","style":{"background":"light","features":[{"name":"roads","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":42,"g":147,"b":187,"a":0.7},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}},{"name":"buildings","enabled":true,"baseEnabled":true,"highlightEnabled":true,"base":{"line-color":{"r":214,"g":85,"b":109,"a":0.3},"line-width":1},"highlight":{"line-color":{"r":255,"g":255,"b":255,"a":0.3},"line-width":1}}]}},"format":"gif","speed":1,"size":"1280x720","dir":"2edc4eb019f8968b12c398a89b4e65e8"}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
Error: Error
    at Actor.receive (/app/node_modules/mapbox-gl/dist/mapbox-gl.js:526:894)
 {}
info: electron process: 2018-07-17 16:00:00 (0/7 [days])
 {}
info: electron process: 2018-07-17 16:00:00 (0/7 [days])
 {}
info: electron process: 2018-07-18 16:00:00 (1/7 [days])
 {}
info: electron process: 2018-07-19 16:00:00 (2/7 [days])
 {}
info: electron process: 2018-07-20 16:00:00 (3/7 [days])
 {}
info: electron process: 2018-07-21 16:00:00 (4/7 [days])
 {}
info: electron process: 2018-07-22 16:00:00 (5/7 [days])
 {}
info: electron process: 2018-07-23 16:00:00 (6/7 [days])
 {}
info: electron process: 2018-07-24 16:00:00 (7/7 [days])
 {}
info: electron process: debug: rendering finished {}
 {}
debug: electron rendering finsished {}

This one finished despite errors. Here's the gif:

[dakotabenjamin]

So it doesn't seem to be writing the buildings and roads:

[szymonkaliski]

This is very strange:

1. does it happen on master?
2. can you reproduce on a local machine, or only on AWS?

[dakotabenjamin]

I think @mataharimhairi was experiencing the same issue on master and locally.