Difference between revisions of "Buildbot Network"

From MythTV Official Wiki
Jump to: navigation, search
 
(17 intermediate revisions by 5 users not shown)
Line 2: Line 2:
 
[[User:Stuarta|stuarta]] 13:35, 3 March 2014 (UTC)
 
[[User:Stuarta|stuarta]] 13:35, 3 March 2014 (UTC)
  
MythTV uses an extensive set of buildslaves, running different operating systems, different versions of Qt (Qt4 & Qt5) as well as different compilers. The aim of this is to validate all changesets do not affect our ability to build on the different platforms.
+
MythTV uses an extensive set of buildworkers, running different operating systems, as well as different compilers. The aim of this is to validate all changesets do not affect our ability to build on the different platforms.
  
 
The intention is to ensure that MythTV will build and run on all the current distro flavours, and in particular on the long term stable variants. eg.
 
The intention is to ensure that MythTV will build and run on all the current distro flavours, and in particular on the long term stable variants. eg.
Line 9: Line 9:
 
* Ubuntu LTS
 
* Ubuntu LTS
 
* Fedora
 
* Fedora
 +
* FreeBSD
 +
* OS X
 
* RHEL/Centos6 is not an option due to the Qt version being too old.
 
* RHEL/Centos6 is not an option due to the Qt version being too old.
  
=== Build Master ===
+
Architecture Support:
  
You can look at the progress of current builds or the history of the most recent builds by visiting [https://code.mythtv.org/buildbot BuildMaster]. I'd recommend either the waterfall or console pages.
+
* i386
 +
* x86_64
 +
* arm (currently armhf, armv6, armv7hl)
  
=== Setting up a build slave ===
+
== Build Master ==
  
Setting up a build slave is a relatively simple process. If you have a server capable of running a virtual machine or three, then offers of buildslaves are always welcome. Please contact the developers via IRC #mythtv on freenode.
+
You can look at the progress of current builds or the history of the most recent builds by visiting [https://code.mythtv.org/buildbot BuildMaster]. I'd recommend the console page.
  
==== Basic VM configuration ====
+
== Setting up a build worker ==
 +
 
 +
Setting up a build worker is a relatively simple process. If you have a server capable of running a virtual machine or three, then offers of buildworkers are always welcome. Please contact the developers via IRC #mythtv on freenode.
 +
 
 +
=== Basic VM configuration ===
  
 
* 1x CPU
 
* 1x CPU
* 2Gb RAM
+
* 2Gb RAM min
* 8Gb OS Disk
+
* 16Gb OS Disk
 
* 50Gb /home
 
* 50Gb /home
  
 
Notes:
 
Notes:
* 2GB of RAM is required as tv_rec.cpp can take an extremely long time to build and/or timeout the build when running only 1Gb ram
 
 
* 50Gb /home is sufficient to hold at least 3 different builds as well as the whole 10Gb ccache
 
* 50Gb /home is sufficient to hold at least 3 different builds as well as the whole 10Gb ccache
  
==== Installation ====
+
== Installation ==
  
Install the operating system you are planning to use, along with all the required packages. Here are a few recipies for setting up buildslaves
+
Install the operating system you are planning to use.
  
* [[Buildslave Debian]]
+
=== Automated Method (Recommended) ===
* [[Buildslave Ubuntu]]
 
* [[Buildslave Fedora]]
 
  
You should also create a buildslave user, and configure ccache to be allowed to use 10Gb of cache '''ccache -M 10G'''
+
(For CentOS first install epel-release)<br>
 +
Install the ansible, ansible-extras (if a separate package on your distro) and git packages, then clone the MythTV ansible repository
  
==== Connecting to the build network ====
+
    $ git clone https://github.com/MythTV/ansible.git
  
For this you will need to talk to the developers on IRC again. You should create a passwordless ssh key for your buildslave user, this will need to be added to allow the builder to checkout the code from our git repository.
+
Then as root, change into the directory where the playbooks were checked out to and run
  
You will be provided with the following pieces of information
+
    # ansible-playbook -i hosts buildworker.yml
  
* your buildslave's name
+
This will install all the packages required, as well as configuring ccache and setting up the startup script to start the buildworker when the server starts. All that is left to do is configure the buildworker. Continue from the step "Connecting to the build network"
* the password assigned to your buildslave
 
* the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007
 
  
==== Create the buildslave ====
+
=== Manual Method ===
  
run the 'buildslave' command as follows
+
Here is some information on setting up buildworker by hand
  
  buildslave create-slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD
+
* [[Buildslave Debian]]
 +
* [[Buildslave Ubuntu]]
 +
* [[Buildslave Fedora]]
  
I suggest using mythtv as your BASEDIR. Please make sure you edit info/host and info/admin in your BASEDIR so we know some information about your buildslave and your contact details
+
You should also create a buildworker user, and configure ccache to be allowed to use 10Gb of cache '''ccache -M 10G'''
  
Then a build can be forced by talking to MythBuild on irc in #mythtv-commits
+
==== Autostart on boot ====
 
 
  MythBuild: force build <buildslave_name> <message>
 
  
==== Autostart on boot ====
+
''NOTE: Only required if you used the manual installation method''
  
Configuring the buildslave to autostart on system boot is a very simple step. Create a script '''~/bin/start-buildslave.sh''' with the following contents
+
Configuring the buildworker to autostart on system boot is a very simple step. Create a script '''~/bin/start-buildworker.sh''' with the following contents if you are using buildbot < 0.9
  
 
   #!/bin/bash
 
   #!/bin/bash
Line 70: Line 74:
 
   /usr/bin/buildslave stop
 
   /usr/bin/buildslave stop
 
   exec /usr/bin/buildslave start
 
   exec /usr/bin/buildslave start
 +
 +
Or this if you are using buildbot >= 0.9
 +
 +
  #!/bin/bash
 +
  cd ~/mythtv
 +
  /usr/bin/buildbot-worker stop
 +
  exec /usr/bin/buildbot-worker start
  
 
Notes:
 
Notes:
* The stop is not a mistake. The stop causes the buildslave to cleanup any old PID files that are present which would otherwise prohibit the buildslave from starting
+
* The stop is not a mistake. The stop causes the buildworker to cleanup any old PID files that are present which would otherwise prohibit the buildworker from starting
 +
 
 +
Then '''chmod +x ~/bin/start-buildworker.sh'''.
 +
 
 +
Next edit crontab to start it at boot time. As the buildworker user run '''crontab -e''' and add the following line
 +
 
 +
  @reboot /home/buildworker/bin/start-buildworker.sh
 +
 
 +
 
 +
== Connecting to the build network ==
 +
 
 +
For this you will need to talk to the developers on IRC again.
 +
 
 +
You will be provided with the following pieces of information
 +
 
 +
* your buildworker's name
 +
* the password assigned to your buildworker
 +
* the hostname and port number of the buildmaster, i.e. buildbot.mythtv.org:9989
 +
 
 +
=== Configure the buildworker ===
 +
 
 +
Run the 'buildworker' command as follows for buildbot < 0.9
 +
 
 +
  # su - buildworker
 +
  $ mkdir mythtv
 +
  $ buildslave create-slave mythtv buildbot.mythtv.org:9989 SLAVENAME PASSWORD
 +
 
 +
Or like this for buildbot >= 0.9
 +
 
 +
  # su - buildworker
 +
  $ mkdir mythtv
 +
  $ buildbot-worker create-worker mythtv buildbot.mythtv.org:9989 WORKERNAME PASSWORD
  
Then '''chmod +x ~/bin/start-buildslave.sh'''.
+
Please make sure you edit info/host and info/admin in ~buildworker/mythtv so we know some information about your buildworker and your contact details.
 +
NOTE: The startup script installed by the automated method will automatically populate info/host at startup
  
Next edit crontab to start it at boot time. As the buildslave user run '''crontab -e''' and add the following line
+
buildworker needs to be started if not already running by either reboot, or by running the following as the buildworker user
  
  @reboot /home/buildslave/bin/start-buildslave.sh
+
    ~/bin/start-buildworker.sh

Latest revision as of 13:50, 27 August 2020

Buildbot Network

stuarta 13:35, 3 March 2014 (UTC)

MythTV uses an extensive set of buildworkers, running different operating systems, as well as different compilers. The aim of this is to validate all changesets do not affect our ability to build on the different platforms.

The intention is to ensure that MythTV will build and run on all the current distro flavours, and in particular on the long term stable variants. eg.

  • Debian Stable
  • Ubuntu LTS
  • Fedora
  • FreeBSD
  • OS X
  • RHEL/Centos6 is not an option due to the Qt version being too old.

Architecture Support:

  • i386
  • x86_64
  • arm (currently armhf, armv6, armv7hl)

Build Master

You can look at the progress of current builds or the history of the most recent builds by visiting BuildMaster. I'd recommend the console page.

Setting up a build worker

Setting up a build worker is a relatively simple process. If you have a server capable of running a virtual machine or three, then offers of buildworkers are always welcome. Please contact the developers via IRC #mythtv on freenode.

Basic VM configuration

  • 1x CPU
  • 2Gb RAM min
  • 16Gb OS Disk
  • 50Gb /home

Notes:

  • 50Gb /home is sufficient to hold at least 3 different builds as well as the whole 10Gb ccache

Installation

Install the operating system you are planning to use.

Automated Method (Recommended)

(For CentOS first install epel-release)
Install the ansible, ansible-extras (if a separate package on your distro) and git packages, then clone the MythTV ansible repository

   $ git clone https://github.com/MythTV/ansible.git

Then as root, change into the directory where the playbooks were checked out to and run

   # ansible-playbook -i hosts buildworker.yml

This will install all the packages required, as well as configuring ccache and setting up the startup script to start the buildworker when the server starts. All that is left to do is configure the buildworker. Continue from the step "Connecting to the build network"

Manual Method

Here is some information on setting up buildworker by hand

You should also create a buildworker user, and configure ccache to be allowed to use 10Gb of cache ccache -M 10G

Autostart on boot

NOTE: Only required if you used the manual installation method

Configuring the buildworker to autostart on system boot is a very simple step. Create a script ~/bin/start-buildworker.sh with the following contents if you are using buildbot < 0.9

  #!/bin/bash
  cd ~/mythtv
  /usr/bin/buildslave stop
  exec /usr/bin/buildslave start

Or this if you are using buildbot >= 0.9

  #!/bin/bash
  cd ~/mythtv
  /usr/bin/buildbot-worker stop
  exec /usr/bin/buildbot-worker start

Notes:

  • The stop is not a mistake. The stop causes the buildworker to cleanup any old PID files that are present which would otherwise prohibit the buildworker from starting

Then chmod +x ~/bin/start-buildworker.sh.

Next edit crontab to start it at boot time. As the buildworker user run crontab -e and add the following line

 @reboot /home/buildworker/bin/start-buildworker.sh


Connecting to the build network

For this you will need to talk to the developers on IRC again.

You will be provided with the following pieces of information

  • your buildworker's name
  • the password assigned to your buildworker
  • the hostname and port number of the buildmaster, i.e. buildbot.mythtv.org:9989

Configure the buildworker

Run the 'buildworker' command as follows for buildbot < 0.9

  # su - buildworker
  $ mkdir mythtv
  $ buildslave create-slave mythtv buildbot.mythtv.org:9989 SLAVENAME PASSWORD

Or like this for buildbot >= 0.9

  # su - buildworker
  $ mkdir mythtv
  $ buildbot-worker create-worker mythtv buildbot.mythtv.org:9989 WORKERNAME PASSWORD

Please make sure you edit info/host and info/admin in ~buildworker/mythtv so we know some information about your buildworker and your contact details. NOTE: The startup script installed by the automated method will automatically populate info/host at startup

buildworker needs to be started if not already running by either reboot, or by running the following as the buildworker user

   ~/bin/start-buildworker.sh