• The 00check.log file of the check output.
• The 00install.out file, the output of the installation of the package. This is often helpful if the installation fails, as the normal check output omits the details in this case.
• Output of the tests.
• Source R package. This should be the same as the uploaded one.
• Binary R package for your platform.
• Binary packages for all dependent packages that were built from source. These include packages specified via the Remotes mechanism. See more about that here.

I reorganized the rhub package, and created R6 classes to hold information about the checks. Much cleaner now. Functions check(), check_for_cran() list_my_checks(), list_package_check(),e etc. all return rhub_check objects.

There is also a new function last_check() that returns the last submitted check of the current R session.

Bug fixes

• The web submission checks the submitted filename. If it does not look like a proper R source package, built via R CMD build, then we reject it immediately.

• On Linux platforms, we are now installing pandoc from the binaries built by RStudio. Unfortunately this still fails sometimes: https://github.com/r-hub/rhub-linux-builders/issues/10 https://github.com/r-hub/rhub-linux-builders/issues/15 We will have to build our own pandoc binaries or packages…

• qpdf is needed for checking PDF documentation. It is now installed on the Linux builders. Windows coming soon.

• The Jenkins job reaper does not fail on unusual jobs, e.g. ones that were aborted or that never ran for some reason.

• Set _R_CHECK_FORCE_SUGGESTS_ to false on Windows, by default.

• Allow overriding R_CHECK_FORCE_SUGGESTS_ on Linux. (Previously it was set after the user supplied environment variables.)

• Use our CRAN mirror on Windows as well. For this I had to make it public, because the windows builders are classic Azure machines, so they are on a different internal network. I also installed HTTPS on it, via letsencrypt.

• Fixed the Jenkins job reaper, the rhub-cron process that removes old jobs from Jenkins.

• Updated the R-hub API, so that a job can be submitted for multiple platforms. This is needed for the check_for_cran() function in rhub.

• Added the check_for_cran() function to the rhub package. It runs the checks on 3-4 platforms, depending on whether the package has compiled code.

Nice emails.

The email sending happens on the front-end app now, Jenkins only sends a signal that the build is done. This gives us much more flexibility in formatting the emails.

Log database

The check logs are stored in a log database. This allows a better API, e.g. querying checks for an email address.

New API endpoints

• Better status API, using the log database, it handles multiple checks at once.
• livelog to supply the build log in a non-stream format.

rhub package improvements

• status API, to query the status of a build. It also handles multiple checks.
• list API, to list checks for an email address and/or a package.
• Cache id of last submission. So the status can be checked easily.
• Platform chooser. This is used by default if no platform is specified and the R session is interactive.
• Revamped the email validation API.

Smaller changes and Bug fixes

• Fixed the f90 compiler on the Debian builders. We need to set the FC option when compiling R.

• Created a repository with small test packages, and known check output: https://github.com/r-hub/test-packages

• We use our internal CRAN mirror, to avoid external traffic, and speed up the check. Both on Linux and Windows.

• Avoid streaming the Jenkins log, this gives a nicer API for the R client. For example it avoids splitting into lines in R. This happends in JS now, and it is more reliable. It also reduces the traffic.

• Handle ABORTED builds. These are typically timeouts because of no output.

• Set _R_CRAN_FORCESUGGESTS to false on all builders, but allow overriding it.

• Fixed the Jenkins job reaper that runs every hour and removes jobs older than three days.

It is ten minutes without output currently. The time limit plugin does not allow me to set an additional hard limit as well…

Passing arguments to R CMD check

Works now on Linux. Need to write some PowerShell to make it work on Windows: https://github.com/r-hub/wincheck/issues/6

Docker cleanup on the image builder

The docker image builder cleans up old images and containers now.
The builders should do the same, ideally.

Installing valgrind

On all ~15 Linux images, to make sure it works.

rhub package fixes:

• Allow the email tokens from the web app in the CLI app as well: https://github.com/gaborcsardi/rhub/issues/8 The other way does not work currently.

• Checking tarballs works now: https://github.com/gaborcsardi/rhub/issues/14

• SAN runs work fine now: https://github.com/r-hub/rhub-backend/issues/10 https://github.com/gaborcsardi/rhub/issues/9

• I checked that the package works fine on Windows / OSX / Linux. We’ll have proper CI at some point…

• Wrote a bunch of test cases. Found and fixed a couple of small bugs as well. https://github.com/gaborcsardi/rhub/issues/24

• Wrote higher level wrapper functions, so one can just say check_on_windows() instead of selecting the exact platform. https://github.com/gaborcsardi/rhub/issues/20 https://github.com/gaborcsardi/rhub/issues/22

• One can pass R CMD check arguments, e.g. --valgrind to the builder, and also environment variables. https://github.com/gaborcsardi/rhub/issues/4

Windows

• There is a nice image now, called win-builder-2016-10-03 that has everything installed, hopefully.

• The image has r-oldrel, r-release, r-patched and r-devel installed, and the frontend and backend also has everything needed to support them.

HTTP/2

• It turns out that the stable nginx version (0.10.x) does not handle http/2 properly, so I had to turn it off. More details: https://github.com/gaborcsardi/rhub/issues/19 https://github.com/curl/curl/issues/1040 https://github.com/dokku/dokku/issues/2435

So currently I still use nginx 0.10.x with http/2 turned off.

Running Jenkins in a Docker container was not the best idea, in the sense that it was quite fiddly to manage. So I’ll install Jenkins now on a new VM, without Docker.

Installing Jenkins

Create a new VM on Amazon. I used an Ubuntu 16.04 HVM AMI. The instructions are for older Ubuntu versions I guess, because they mention openjdk-7-jre, but I suppose, the newer Java should work fine.

Follow https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu:

1
2
3

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y openjdk-8-jre openjdk-8-jdk

1
2
3
4

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install -y jenkins

Need to change the jenkins startup options in /etc/default/jenkins to make sure Jenkins stores files in UTF-8, because it seems to ignore the system locale:

1

JAVA_ARGS="-Dfile.encoding=UTF-8 ..."

The same flag must be added to the script that starts the workers, see later.

Add/redirect DNS

Set the A record of jenkins.r-hub.io to the new server.

Configure Jenkins

Make sure you have access to port 8080 on the instance, and then start configuring Jenkins from your browser.

Install no plugin from the installer. It does not have the full list,
so we’ll have to install some plugins later, anyway. So we’ll do it all at once.

Create an admin user called admin, and a secure password.

Install the following plugins using the plugin manager:

• Build timeout plugin
• Cross-platform shell plugin
• Email extension plugin
• Elastic axis plugin
• Groovy postbuild plugin
• Mailer plugin
• PowerShell plugin
• Self-Organizing Swarm Plug-in Modules
• SSH Slaves plugin
• Workspace cleanup plugins

Then configure Jenkins:

• Set the quiet period to 0, no need to wait before starting a build.
• Setup global environment variables: PS4="+R-HUB-R-HUB-R-HUB"
• Jenkins URL: http://jenkins.r-hub.io:8080/
• Admin email address: admin@r-hub.io
• Extended email notification requires some configuration:
  1. SMTP server: 127.0.0.1
  2. Default User email suffix: @r-hub.io
  3. Reply-to list: admin@r-hub.io
• In Jenkins global security config, set the TCP port for JNLP agents to fixed 50000.
• Set the Security realm to Jenkins’ own user database, do not allow users to sign up
• Set the authorization to Matrix-based security, give overall Administter rights to admin
• Turn off the “Prevent Cross Site Request Forgery exploits” switch. This does not work currently, I am not sure why.
• Turn on Slave → Master Access Control

Install and configure mail server

1

apt-get install postfix

The name of the system is jenkins.r-hub.io.

Add workers

SSH to a Linux worker, and restart the swarm process:

1

restart swarm JENKINS_PASSWORD=<passwd> RHUB_IP=jenkins.r-hub.io

On the windows workers, just restart the Java process with the new Jenkins server

Quite annoyingly, Azure sometimes changes the IP address of the builders, and then I need to update the jenkins security group on AWS.

Update builder config

Make sure that the Jenkins server, and the builder server (that runs the frontend and the backend) are both have the same (default) security group.

Configure the JENKINS_URL environment variable on dokku, use the internal IP address of the Jenkins server. E.g.

1

dokku config:set cron JENKINS_URL=http://admin:<passwd>@172.31.30.118:8080

Do this for all node apps that need access to Jenkins: builder, buidler-test, backend and cron currently.

OK, for the Sensu addition yesterday it turns out that two Sensu
plugins are also needed on clients:

1
2

sensu-install -p cpu-checks
sensu-install -p disk-checks

and then the new Sensu client is all green.

Getting rid of a Sensu check

This is surprisingly difficult. Yesterday I switched cranlogs.r-pkg.org to HTTPS, after getting a new certificate for it. It seems to work fine now, I don’t know what was the problem last time (https://github.com/metacran/cranlogs.app/issues/22), but it works fine today.

I also updated the Sensu checks for cranlogs, to check HTTPS and also the HTTP -> HTTPS redirect. It seems that to get rid of the old check, one has to restart the Sensu server and the client as well, both running on monitor.r-hub.io in this case.

Also, one needs to put the port number into the check. Even if the nginx redirect header does not have it: https://cranlogs.r-pkg.org:443/downloads/daily/last-week/igraph

Azure disk attach/detach

This seems to a straightforwars enough operation, but it is not. Make sure that you close the browser tab, and open a new one to Azure, after each detach/attach operation, because the UI does not refresh the state of the disk. It is actually the same for the state of the virtual machine, you click on Stop, it stops (after some time), but the state is never updated, until a new browser UI window is used.

I wonder who thought this was a good idea. It only took me ~4 hours to solve this, thank you.

Fedora update

Finally I fixed the Docker builds in cran.r-hub.io, and built new Fedora images. (https://github.com/r-hub/rhub-linux-builders/pull/12#issuecomment-247629893)

In the process of brushing up Jenkins, I made sure that I get emails about failures, both from cran.r-hub.io and jenkins.r-pkg.org. For this one needs to install postfix, and on jenkins.r-pkg.org, one also needs to allow the Docker subnets in the main.mf postfix config file, because there Jenkins runs in a container.

The only glitch is that the clang image does not build, because I would need another external clang repository for that. :(

Dokku update on metacran

I updated Dokku to the latest stable version (0.7.1) on r-pkg.org. This affects cranlogs.r-pkg.org, rversions.r-pkg.org, crandeps.r-pkg.org as well. The new version uses the new plugins for Redis and PostgreSQL as well.

OS updates

First of all, I ran apt-get update; apt-get upgrade etc. on the various hosts:

• r-hub.io
• ci.r-hub.io
• Linux builder 1
• Linux builder 2
• cran.r-hub.io
• monitor.r-hub.io
• install-github.me
• jenkins.r-pkg.org
• seer.r-pkg.org
• crandb.r-pkg.org
• docs.r-pkg.org
• www.r-pkg.org

AWS changed the IP address of ci.r-hub.io, because I stopped it. Hmmm, it really should not do that. I had to change it in the DNS.

Running out of disk

The disk of cran-r-hub.io was filling up. This is the machine that builds the Docker images for the builders, so it needs more disk as we are adding more images. It is easy to resize the disk on Azure, just:

• shut down the machine,
• stop it from the Azure web portal (wait for it…),
• resize the disk on the Azure web portal,
• start the machine.
  Resized to 100GiB now, this should be enough for a lot of images.

SSL certificates

Next, I updated the SSL certificates for

• ci.r-hub.io
• cranlogs.r-pkg.org

Monitoring

I added monitoring to the following machine:

• cran.r-hub.io

The following are monitored:

• whether the machine is up,
• disk space,
• cpu load

For Linux machines, I had to run this. Add repos and install packages first:

1
2
3
4

wget -q https://sensu.global.ssl.fastly.net/apt/pubkey.gpg -O- | sudo apt-key add -
echo "deb https://sensu.global.ssl.fastly.net/apt sensu main" | sudo tee /etc/apt/sources.list.d/sensu.list
sudo apt-get update
sudo apt-get install sensu

Put the following into /etc/sensu/conf.d/rabbitmq.json:

1
2
3
4
5
6
7
8
9
10
11
12
13

{
  "rabbitmq": {
    "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
    "host": "monitor.r-hub.io",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "<omitted>"
  }
}

Copy the Sensu certificate and the private key from the monitoring server to /etc/sensu/ssl/cert.pem and /etc/sensu/ssl/key.pem:

1
2
3

scp ubuntu@monitor.r-hub.io:/etc/sensu/ssl/cert.pem .
scp ubuntu@monitor.r-hub.io:/etc/sensu/ssl/key.pem .
...

Make sure they are owned by sensu:sensu.

Put this into /etc/sensu/conf.d/client.json:

1
2
3
4
5
6
7

{
  "client": {
    "name": "cran.r-hub.io",
    "address": "13.88.23.239",
    "subscriptions": [ "common" ]
  }
}

Enable sensu on boot, start the client:

1
2

sudo update-rc.d sensu-client defaults
sudo /etc/init.d/sensu-client start

Enable the client in the AWS firewall, in the sensu dashboard security group.

Lesson learned: in PowerShell, if you WaitForExit on a Process, then the no output events are generated until it will be finished. I guess it should be named WaitForExitAndDoNothingElse… anyway, this is fixed now in this commit.

Another painful lesson is that apparently not everything is included in a generalized image on Azure. I did install Carbon, and more PowerShell stuff before capturing the image, but they were not present in the image. :( I guess I need to install them to some system directory, instead of the user’s home.

Now I need to section the output on Windows, this issue), and then make the output nicer

• in the rhub client,
• on the web,
• in the email message.

The main glitch is that the output is not streamed. This will require some more PowerShell tricks to fix. The problem is that the output is coming from another process, ran by another user, and this is apparently non-standard.

A bigger issue will be to work out how we should use sysreqs on Windows.

• linux-x86_64-centos6-epel
• linux-x86_64-centos6-epel-rdt
• linux-x86_64-rocker-gcc-san

The last one runs the check with address sanitizer turned on. Unfortunately the ASAN/UBSAN output is suppressed currently,
so we still need to work on this. :(

Some lessons learned, and improvements:

• The images must have curl, this is there on the CentOS6 images by default, but not on the Rocker images.
• The images must have xvfb as well.
• We are running xvfb now with a 1024x768 screen, and 24 bits color depth. This eliminates some R problems, that come up when not enough colors are available.
• The images may set the RBINARY environment variable. The ASAN image uses this to start the ASAN-augmented version of R, which is called RD on the image.
• Jenkins now starts bash in the Docker container with -l, to make it a login shell. This allows running /etc/bashrc in the image. The epel-rdt image runs the RDT config file this way.

System requirements improvements

We have also added a bunch of new system requirements.

It’s UI needs a lot of improvements, but the streaming part works fine,
thanks to the streaming support of the curl R package.

I am getting closer to supporting windows builds. I started to write two Powershell scripts, that will eventually perform the windows builds on the slaves: https://github.com/r-hub/wincheck

The first one run.ps1

• creates a new user with a random username,
• creates its home directory as well,
• makes sure that the home directory is not readable by other users,
• starts the build process (other script) as the newly created user, and makes sure that its standard output and error are redirected and still printed to the screen.

The other script is slave.ps1. This runs the actual R CMD check. It all works now, but I need more features, e.g. installing dependencies. In general it should do everything the unix version (at https://github.com/r-hub/rhub-backend/blob/master/templates/jenkins.sh) does.

Eventually these powershell scripts should go in the rhub-backend repository I believe.

Somewhat surprisingly, Powershell is not very good at some sysadmin tasks, e.g. creating local users is hard in plain Powershell. So for this we require Carbon (http://get-carbon.org/) for now, this has to be installed on the Windows builders.

• r-patched-linux-x86_64 Debian testing
• r-devel-linux-x86_64-ubuntu-16.04 most recent Ubuntu LTS
• r-release-linux-x86_64-ubuntu-16.04 most recent Ubuntu LTS
• r-devel-linux-x86_64-debian-jessie stable Debian
• r-release-linux-x86_64-debian-jessie stable Debian

Adding new images is somewhat complicated. I need to add them at several places:

• Add the Dockerfile to the rhub-linux-builders repo.
• Create a repository on Docker Hub.
• Add an entry to jenkins.sh in the same repo, to build it periodically.
• Add an entry to public/data/platforms.json on the rhub-frontend repo.
• Add it to the Jenkins job that pulls images to the workers.
• Run the Jenkins job that builds the images, and/or build them by hand and push them to Docker Hub
• Add the platform to the sysreqs database, if it is not there yet. Update the database from GitHub.

Then one needs to re-reploy the frontend (!).

We need some central configuration store, like etcd, or something similar, to avoid this lengthy and error-prone procedure.

Windows builders

Jenkins has an Azure plugin. Unfortunately it uses the ‘classic’ system, which is soft-deprecated on Azure, and the branch to update it to the new Azure Resource Manager seems to be dead.

After a couple of hours I could get it work. Key points:

• Use the JNLP launch method. Much easier than SSH.
• The JNLP port needs to be fixed to 50000, but it was already fixed, anyway.
• Leave the 8080 and 50000 ports open on the master. These must be restricted to the slave IP addresses in the future.
• A new user ‘slave’ is needed in Jenkins, and its name and token must be added to the example init file.
• The new user needs capabilities: overall/read, slave/connect and slave/disconnect.

As for provisioning, I’ll write a short powershell script that downloads the real provisioning script from GitHub, similarly to how r-appveyor works.

Seems like Jenkins does not drop TCP connections coming for the builder web app to stream the console output. What is even worse, it keeps a thread for each connection, and gets really slow already around ~20 threads. (What the hell is it doing? It should be just listening on a socket….)

Anyway, this needs to be fixed: https://github.com/r-hub/rhub-frontend/issues/8

cron app dies sometimes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

2016-07-04T10:45:10.973610931Z app[web.1]: > rhub-cron@1.0.0 start /app
2016-07-04T10:45:10.973617561Z app[web.1]: > node ./bin/scheduler
2016-07-04T10:45:10.973623287Z app[web.1]:
2016-07-04T11:42:42.620726576Z app[web.1]: /app/index.js:33
2016-07-04T11:42:42.620802676Z app[web.1]: 	jen.build.get(job.name, data.lastBuild.number, function(err, data) {
2016-07-04T11:42:42.620850863Z app[web.1]: 	                                      ^
2016-07-04T11:42:42.620885526Z app[web.1]:
2016-07-04T11:42:42.620919177Z app[web.1]: TypeError: Cannot read property 'number' of null
2016-07-04T11:42:42.620927075Z app[web.1]:     at /app/index.js:33:40
2016-07-04T11:42:42.620932178Z app[web.1]:     at next (/app/node_modules/papi/lib/client.js:292:32)
2016-07-04T11:42:42.620946196Z app[web.1]:     at Jenkins.body (/app/node_modules/jenkins/lib/middleware.js:14:3)
2016-07-04T11:42:42.620950958Z app[web.1]:     at next (/app/node_modules/papi/lib/client.js:298:10)
2016-07-04T11:42:42.620955722Z app[web.1]:     at Jenkins.<anonymous> (/app/node_modules/jenkins/lib/middleware.js:88:5)
2016-07-04T11:42:42.620960668Z app[web.1]:     at next (/app/node_modules/papi/lib/client.js:298:10)
2016-07-04T11:42:42.620965208Z app[web.1]:     at Jenkins._onResponse (/app/node_modules/jenkins/lib/jenkins.js:109:3)
2016-07-04T11:42:42.620981759Z app[web.1]:     at next (/app/node_modules/papi/lib/client.js:298:10)
2016-07-04T11:42:42.620988281Z app[web.1]:     at IncomingMessage.<anonymous> (/app/node_modules/papi/lib/client.js:598:7)
2016-07-04T11:42:42.621025915Z app[web.1]:     at emitNone (events.js:85:20)

https://github.com/r-hub/rhub-cron/issues/1

Upload storage, dokku upgrade

Uploads must be moved out of the container, because if I update the web app, they are deleted. Dokku makes this easy with the new storage plugin (http://dokku.viewdocs.io/dokku/dokku-storage/), but for this I had to upgrade dokku to 0.6.4.

It all seems to work fine, except that every time I redeploy the frontend, I usually need to redeploy the backend as well….

Closes https://github.com/r-hub/rhub-frontend/issues/7

Periodically building the docker images

For the devel R versions at least. I cannot do this on Docker Hub, because I have multiple images in a single repo (I really do not want 20 repos for 20 images.) I also cannot do it on Travis, because with 6 images I already hit the 50 minutes time limit.

So I’ll do it on the Jenkins CI on Azure (cran.r-hub.io). It takes about 2 hours to build them now. I’ll do them once a week (Sunday evening), and also whenever the repo with the Dockerfiles changes. I have added a hook for this. The job pushes the images to Docker Hub.

I still need to have a way to pull them to the worker nodes, I guess that’ll be just another Jenkins job on the workers. This one can run every hour, if there are no new images, then it is quick. Also, just one job is enough, just need to configure it to run on all workers.

Apparently, I need the Elastic Axis plugin for this. I run the pulls sequentially, so that the worker nodes don’t update at the same time.

Use a single keepalive job

Instead of one per worker. Now that I have found the Elastic Axis plugin.

Test builder web app

Is at http://builder-test.r-hub.io. Note the http, letsencrypt seems to be down, so I could not get a certificate. This has a different redirect URL, so it needs a separate GitHub application.

Alternative email address

https://github.com/r-hub/rhub-frontend/issues/9 The test builder uses this now.

About this log

This log serves as a notebook for what is being done on r-hub. I probably will not make an effort to make it very readable, and it will be mostly technical.