GPL16043 Upload Job

[jdimatteo]

Charles explained the following in an email:

the user uploads a zip file and provides a name

jenkins can then call the following job

/ark/home/cl512/pipeline/GPL16043.sh ARG1 ARG2

e.g.
bash /ark/home/cl512/pipeline/GPL16043.sh /ark/home/cl512/ressrv19/raw/expression/JB20130926st.zip affy2

where ARG1 is the path to the zip file

and ARG2 is the name that the user provided (the name can't have any spaces)

In the last line of the std out of GPL16043.sh, the location of the output file is provided as follows (example):

"WROTE OUTPUT TO: /ark/temp/affy2_tmp_15578/affy2_output.zip"

this zip should then be sent by email to the user.

[jdimatteo]

I updated the permissions to /ark/temp so that any user can add files/folders (and remove/modify files that they own).

jdm@tod:~$ sudo chmod a+rwxt /ark/temp/

This fixed a problem where the jenkins user couldn't add the temp folders needed by GPL16043.sh.

[jdimatteo]

@bradnerComputation / Charles, this is all set up now, let me know if you have any problems or if I should make any changes:

http://tod.dfci.harvard.edu/jenkins/job/Normalize_GPL16043/build?delay=0sec

[charlesylin]

Awesome. Will test tonight. Thanks for your help as always, especially this late.

-Charles

On Oct 17, 2013, at 11:47 PM, jdimatteo notifications@github.com wrote:

@bradnerComputation / Charles, this is all set up now, let me know if you have any problems or if I should make any changes:

http://tod.dfci.harvard.edu/jenkins/job/Normalize_GPL16043/build?delay=0sec

—
Reply to this email directly or view it on GitHub.

[charlesylin]

John,

It seems like the uploaded zip isn't getting cp'd to the right tmp folder

Started by user anonymous http://tod.dfci.harvard.edu/jenkins/user/null
Building in workspace /ark/home/jenkins/jobs/Normalize_GPL16043/workspace
Checkout:workspace /
/ark/home/jenkins/jobs/Normalize_GPL16043/workspace -
hudson.remoting.LocalChannel@e1935d6
Using strategy: Default
Last Built Revision: Revision 79e34215db5a50d8df3a41c9081673d2741d578f
(origin/HEAD, origin/master)
Fetching changes from 1 remote Git repository
Fetching upstream changes from origin
Seen branch in repository origin/HEAD
Seen branch in repository origin/master
Seen 2 remote branches
Commencing build of Revision 79e34215db5a50d8df3a41c9081673d2741d578f
(origin/HEAD, origin/master)
Checking out Revision 79e34215db5a50d8df3a41c9081673d2741d578f
(origin/HEAD, origin/master)
Warning : There are multiple branch changesets here
No emails were triggered.
Copying file to input.zip
[workspace] $ /bin/sh -xe /tmp/hudson3165583539119894779.sh

• rm -f input.zip output.zip
• bash GPL16043.sh input.zip Myeloma
• tee stdout.txt
  cp: cannot stat `input.zip': No such file or directory
  unzip: cannot find or open
  /ark/temp/Myeloma_tmp_11365/Myeloma_tmp_11365.zip,
  /ark/temp/Myeloma_tmp_11365/Myeloma_tmp_11365.zip.zip or
  /ark/temp/Myeloma_tmp_11365/Myeloma_tmp_11365.zip.ZIP.

Charles Y. Lin, Ph.D.
Dana-Farber Cancer Institute
Department of Medical Oncology
charles_lin@dfci.harvard.edumailto:charles_lin@dfci.harvard.edu
http://bradnerlab.com

On Thu, Oct 17, 2013 at 11:52 PM, Charles Lin charles.y.lin@gmail.comwrote:

Awesome. Will test tonight. Thanks for your help as always, especially
this late.

-Charles

On Oct 17, 2013, at 11:47 PM, jdimatteo notifications@github.com wrote:

@bradnerComputation https://github.com/bradnerComputation / Charles,
this is all set up now, let me know if you have any problems or if I should
make any changes:

http://tod.dfci.harvard.edu/jenkins/job/Normalize_GPL16043/build?delay=0sec

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-26570579
.

[jdimatteo]

@bradnerComputation / Charles,

I fixed the input.zip problem, sorry about that.

I still see errors in the stdout though, e.g.

> selibrary(affy)
Error: could not find function "selibrary"
Execution halted

I see this selibrary on the first line of /ark/home/cl512/pipeline/GPL16043.r

Charles, is there a file missing with the definition of selibrary?

Also, on line 28 of GPL16043.sh it is calling "/ark/home/cl512/pipeline/GPL16043.r" instead of just "GPL16043.r" -- was that deliberate? I setup the Jenkins job to check out the latest files from github, so if you just leave off the path GPL16043.sh will use the latest version of GPL16043.r in the git repo instead of whatever is in your home directory.

[charlesylin]

That's a typo that I must have inserted today when I was trying to muck
with the code. I didn't know we could pull right from the github for the
GPL16043.r code. That would definitely be easier.

-Charles

Charles Y. Lin, Ph.D.
Dana-Farber Cancer Institute
Department of Medical Oncology
charles_lin@dfci.harvard.edumailto:charles_lin@dfci.harvard.edu
http://bradnerlab.com

On Fri, Oct 18, 2013 at 9:10 PM, jdimatteo notifications@github.com wrote:

@bradnerComputation https://github.com/bradnerComputation / Charles,

I fixed the input.zip problem, sorry about that.

I still see errors in the stdout though, e.g.

selibrary(affy)
Error: could not find function "selibrary"
Execution halted

I see this selibrary on the first line of GPL16043.r .

Charles, is there a file missing with the definition of selibrary?

Also, on line 28 of GPL16043.sh it is calling
"/ark/home/cl512/pipeline/GPL16043.r" instead of just "GPL16043.r" -- was
that deliberate? I setup the Jenkins job to check out the latest files from
github, so if you just leave off the path GPL16043.sh will use the latest
version of GPL16043.r instead of whatever is in your home directory.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-26640170
.

[jdimatteo]

It seems to be working now that I've made the changes shown below. Let me know if you'd like me to checkin these changes to the repo:

jdm@tod:~/pipeline$ git diff
diff --git a/GPL16043.sh b/GPL16043.sh
index 4058854..83cf604 100755
--- a/GPL16043.sh
+++ b/GPL16043.sh
@@ -12,6 +12,7 @@ ID=$RANDOM
 #ID=1234
 TEMP_DIR_ROOT=/ark/temp/
 TEMP_DIR=$TEMP_DIR_ROOT$NAME\_tmp\_$ID
+INITIAL_DIR=`pwd`

 #making the temp directory
 mkdir $TEMP_DIR
@@ -25,14 +26,14 @@ unzip $TEMP_DIR/$NAME\_tmp_$ID.zip
 mkdir $TEMP_DIR/output

 #run the spikey normy
-R --no-save $TEMP_DIR/ $NAME < /ark/home/cl512/pipeline/GPL16043.r
+R --no-save $TEMP_DIR/ $NAME < $INITIAL_DIR/GPL16043.r

 #run the GPL gene level script
-python /ark/home/cl512/pipeline/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas
-python /ark/home/cl512/pipeline/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas
+python $INITIAL_DIR/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas5_probe_expr
+python $INITIAL_DIR/GPL16043.py -i $TEMP_DIR/output/$NAME\_all_mas5_probe_expr

 #zip up the output
 cd $TEMP_DIR
 zip -r $NAME\_output.zip output

-echo "WROTE OUTPUT TO: " $TEMP_DIR/$NAME\_output.zip
\ No newline at end of file
+echo "WROTE OUTPUT TO: " $TEMP_DIR/$NAME\_output.zip
jdm@tod:~/pipeline$ 

[jdimatteo]

I updated the job to use the fork at https://github.com/jdimatteo/pipeline, which currently just has the changes above. The Jenkins job seems to be working correctly without any errors now