Jenkins

Dashboard

URL: http://jenkins.secondfunnel.com/jenkins/
Default username: user
Default passowrd: willet password

SSH

1. Locate the private key file in Dropbox: /Willet/keys/willet-generic.pem
2. Get the Jenkins Public DNS from in the AWS Console under EC2 Dashboard / Instances (ex: ec2-54-188-22-193.us-west-2.compute.amazonaws.com)
3. SSH using username tomcat, example console command: ssh -i willet-generic.pem tomcat@ec2-54-188-22-193.us-west-2.compute.amazonaws.com
   Note: key must not be publicly viewable for SSH to work. Use this command if needed: chmod 400 willet-generic.pem
4. SSH again using the username bitnami onto jenkins.secondfunnel.com: ssh bitnami@jenkins.secondfunnel.com. As this user, you can run sudo commands.
5. When you SSH into bitnami, you are in /home/bitnami/. Jenkins builds is actually done in /home/tomcat/.jenkins/jobs/__build_name__/workspace/. The build scripts are in Jenkins interface.

Debugging

Build failing

If the error Signature not yet current: YYYYMMDDTxxxxxxZ is still later than YYYYMMDDTxxxxxxZ (YYYYMMDDTxxxxxxZ + 5 min.) appears during a build, run the following commands as a privileged user:

1. ntpq -p to determine if ntp is installed/running on the machine
2. sudo apt-get install ntp if ntp is not installed
3. sudo service ntp stop
4. sudo ntpdate -s ntp.ubuntu.com
5. sudo service ntp start

Thread has died - java.io.IOException: No space left on device

1. df check if a filesystem has run out of memory
2. df -i check if a ton of small files has used up all the inodes
3. sudo du -h /var | grep '[0-9\.]\+G' find the largest (GB) files

If /var/lib/docker/devicemapper/devicemapper is using a lot of space, check how much:

root@ip-10-254-32-156:/# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-202:1-169682-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 2728.2 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 2.8 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.2.0-52-virtual

If Data Space Total is huge, its a known bug. Temporary solution:

bitnami@ip-10-254-32-156:/# sudo service docker stop
docker stop/waiting
bitnami@ip-10-254-32-156:/# sudo rm -rf /var/lib/docker
bitnami@ip-10-254-32-156:/# sudo service docker start
docker start/running, process 19223