New Regression and Plot

[mdsteiner]

Run new mixed effects model (pRisky ~ goalCondition * overGoal + points + (1|id/game)) and create plots with points earned on the x-axis and pRisky on y axis (I think here I will create bins of points to then actually get a probability instead of 0 and 1 for the pRisky).

[mdsteiner]

Here's the plot. It's not super pronounced but I it goes in the direction we'd like:

Regression Analysis

	Estimate	Std. Error	z value	Pr(>z)
(Intercept)	-0.347118	0.062818	-5.526	3.28e-08 ***
goal.condition.fNoGoal	-0.088795	0.084210	-1.054	0.292
overGoal.f1	-0.696863	0.048063	-14.499	< 2e-16 ***
points.cum	-0.002309	0.000292	-7.910	2.57e-15 ***
goal.condition.fNoGoal:overGoal.f1	0.929169	0.057916	16.043	< 2e-16 ***

Tjur's D = 0.232

[ndphillips]

Good Stuff!

First, because I always have trouble interpreting interactions, can you make sure everything is in the correct direction? That is, can you write out the conclusion? Also, could you include environment as an additional main effect?

Second, can you add confidence intervals to the aggregate lines? ggplot would make this very easy ;)

Third, can you create separate plots for the separate conditions? If they all look the same, then we can just what you have. But if they are different, we might want to interpret them.

[mdsteiner]

Plot with SE around mean lines, over all conditions:

Plot with SE around mean lines, separate panels for variance conditions:

That's a really nice thing about ggplot! I already started creating subsets and stuff and only then remembered how easy it is and just added facet_wrap()... Now it looks like in the high and equal environment we see what we like but in the low environment things are a bit fucked up...

About the regression, if I get this stuff correctly it seems that in general, once one is above the 100 points, the probability of selecting the high risk option is lower, but that this is effect is less pronounced in the no goal condition. Also the more points one has earned, the lower the probability of choosing risky (I think this may be due to the weird shapes of the curves we see in the plots, so the relation is not linear but it tries to fit something linear). I will however need to revisit the regression once I'm through with the model recovery (I'm currently working on that) and maybe tweak some things because the model is nearly unidentifiable...

[mdsteiner]

Here is a plot of the likelihood of choosing risky but with Trial instead of Points on the x axis (to show that the above patterns are not solely trial effects):

This is also some kind of sanity check because it shows that in the condition in which the risky option was the better, people s pRisky didn't go down but in the others it did, maybe indicating learning in the low and risk aversion in the equal condition

[ndphillips]

Yep I think you're interpretation of the interaction looks good, but I think the results would be clearer if you recoded the Goal condition predictor as Goal (1 = Yes, 0 = No). That way, we could interpret the Goal (1 = Yes, 0 = No) X OverGoal (1 = Yes, 0 = No) interaction as the change in the likelihood of selecting the risky option once people in the goal condition get over the goal. We would expect this interaction to be strongly negative, as people in the goal condition should become very risk averse once they pass the goal

I think that's a bit clearer than the current interaction which is the change in likelihood of selecting the risky option once people in the No Goal condition get over the goal.

In other words, the default, reference category for the goal condition should be No goal.

[ndphillips]

Here's one other idea: because we mostly predict effects when people are close to the goal, try re-running the regression only for trials where people are within 10 points of the goal. We should expect the size of the interaction between Goal and AboveGoal to be much higher in these cases.

[mdsteiner]

Recoded Regression Analysis

	Estimate	Std. Error	z value	Pr(>z)
(Intercept)	-0.3923004	0.0665101	-5.898	3.67e-09 ***
goal.condition.bin1	0.0865610	0.0960116	0.902	0.367
overGoal.f1	0.2216200	0.0488954	4.533	5.83e-06 ***
points.cum	-0.0026771	0.0003284	-8.151	3.62e-16 ***
goal.condition.bin1:overGoal.f1	-0.9684303	0.0653238	-14.825	< 2e-16 ***

Regression analysis with only last 10 trials

	Estimate	Std. Error	z value	Pr(>z)
(Intercept)	-0.9666009	0.1263716	-7.649	2.03e-14 ***
goal.condition.bin1	0.2581763	0.1666328	1.549	0.121
overGoal.f1	0.3011330	0.0757212	3.977	6.98e-05 ***
points.cum	0.0002105	0.0008785	0.240	0.811
goal.condition.bin1:overGoal.f1	-1.1214398	0.0989148	-11.337	< 2e-16 ***

So the effect of points disappears when we look at the last few trials. Then we only have an effect of being above the goal (means people get riskier) and a bit stronger interaction, meaning that the difference between pRisky between the two goal conditions becomes larger.

Regression analysis with all trials but also environment as predictor

	Estimate	Std. Error	z value	Pr(>z)
(Intercept)	-0.511628	0.076620	-6.677	2.43e-11 ***
goal.condition.bin1	0.112715	0.077717	1.450	0.1470
overGoal.f1	0.231346	0.043935	5.266	1.40e-07 ***
points.cum	-0.002274	0.000292	-7.788	6.81e-15 ***
variance.conditionHigh	0.488517	0.094269	5.182	2.19e-07 ***
variance.conditionLow	-0.307820	0.095010	-3.240	0.0012 **
goal.condition.bin1:overGoal.f1	-0.929184	0.057912	-16.045	< 2e-16 ***

[mdsteiner]

Here I plot the number of trials needed per trial in order to reach the goal on the x axis and the pRisky on the y axis:

Also here we see that the more points per trial are needed, the more risk seeking participants become.

[ndphillips]

Now THAT is a nice looking result!!!! Definitely put that in the results!!!

One issue though is that smoothing splines (which loess uses) aren't very interpretable (ie. they don't have any underlying model)

I would suggest replacing this plot with results from a logistic regression analysis. That is, put points needed per trial on the x-axis and the logit of the likelihood of choosing risky on the y axis (log(p / (1-p)). Then, plot a straight line where the slope is the estimated beta weight from the logistic regression

[ndphillips]

You could also try running another regression analysis with points.needed.per.trialas the main predictor instead of OverGoal and points.cum. If you do, you might also include trial as a covariate.

[mdsteiner]

you are right, I think that makes sense. I will include both models in the manuscript, we can then still decide whether we wanto to keep both of them in there. Also one note about the plots: I included the new plots and will leave them in for a while, for writing and revising the manuscript that shouldn't make a difference, but I will certainly try to reformat them a bit and make them look nicer before we submitt...

[ndphillips]

Yeah I totally agree. don't worry about making the plots perfect for now.