Skip to content

Adding a System to Hardware Continuous Integration

Andrew Geissler edited this page Nov 8, 2021 · 3 revisions

All users are welcome to add their hardware to OpenBMC continuous integration. The only rule is that the owner is responsive to assisting with debug when a failure occurs.

At a high level, this is implemented by the following steps:

  • Monitoring for changes of interest (openbmc/openbmc changes for example)
  • Downloading the flash image generated by the upstream CI job and updating your piece of hardware with it
  • Running a set of tests
  • Updating the corresponding gerrit commit with a pass/fail message

HW CI does not prevent the merging of code (i.e. no actual score on gerrit) but the pass/fail message on the review can be used as guidance by the maintainer who merges the changes.

Special Note on Systems Not a Part of Upstream CI

You can also use the process defined in this wiki for adding your own system to CI. For example, you trigger off of the appropriate meta-* change your system is interested in, and then build and test it within your process.

Example Implementation

Create a github userid for your automation work

For example, IBM uses https://github.com/jenkins-openbmc-ibm. A similar naming convention is recommended (i.e. jenkins-openbmc-<company/function>

Setup a jenkins server to monitor for changes

The easiest way to monitor for changes of interest is to use jenkins. It has a gerrit plugin that can be configured to watch for projects of interest.

  • Follow directions to setup a Gerrit Server within your jenkins instance
  • Create a job that has a "Build Trigger" of Gerrit event and uses a "Gerrit Trigger" which utilize the server you set above
  • Utilize the "Silent Start Mode" option to keep the gerrit noise down and only report the pass/fail at the end
  • "URL to post" should be something like " Hardware CI"
  • Check all "Skip Vote" options because this job does no voting
  • Set "Custom Build Messages" to something relevant like " test passed/failed"
  • "Trigger on" should be when a "Comment Added" has a "Verdict Category" of Verified and a "Value" of 1. Basically, you want to start HW CI once the upstream validation process has completed.
  • "Dynamic Trigger Configuration" is where you put int he repositories you are interested in, like "openbmc/meta-phosphor".

Grabbing the image to test from upstream

Within your jenkins job, you know the gerrit job that you want to test. You need to use this information to find the flash image from the corresponding upstream jenkins job. This is a bit complicated but can be done. Below is some pseudo code which can be used for reference.

#!/bin/bash -xe

echo "Triggered by ${GERRIT_PROJECT}"

# Support both openbmc/openbmc and the new meta layers
if [[ "${GERRIT_PROJECT}" == "openbmc/openbmc" ]]; then
    GERRIT_PATH="ci-openbmc"
    deploy="openbmc/build/tmp/deploy/"
else
    GERRIT_PATH="ci-meta"
    deploy="openbmc/build/tmp/deploy/"
fi

# Get a crumb so we don't have CSRF issues
wget --auth-no-challenge --user <your users secret info> --output-document ${WORKSPACE}/crumb 'https://jenkins.openbmc.org/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)'

url=$(curl --header $(cat ${WORKSPACE}/crumb) -G -s "https://jenkins.openbmc.org/job/$GERRIT_PATH/api/xml" --data-urlencode "tree=builds[number,url,actions[parameters[name,value]]]" --data-urlencode "xpath=/matrixProject/build[action/parameter[name='GERRIT_REFSPEC' and value='${GERRIT_REFSPEC}']][1]/url")
url=${url#"<url>"}
url=${url%"</url>"}

# Note the hard coded "witherspoon" below, this is the system of interest in this example
# Also not the searching for obmc-phosphor-image-witherspoon.ubi.mtd.tar since that is what is needed by the HW CI job in this case

filename=$(curl --header $(cat ${WORKSPACE}/crumb) -G -s "https://jenkins.openbmc.org/job/$GERRIT_PATH/distro=ubuntu,label=docker-builder,target=witherspoon/api/xml" --data-urlencode "tree=builds[actions[parameters[name,value]],artifacts[fileName]]" --data-urlencode "xpath=(matrixConfiguration/build[action[parameter[name='GERRIT_REFSPEC' and value='${GERRIT_REFSPEC}']]]/artifact/fileName[contains(text(),'obmc-phosphor-image-witherspoon.ubi.mtd.tar') and not(contains(text(),'.static.mtd.all.tar'))])[1]")
filename=${filename#"<fileName>"}
filename=${filename%"</fileName>"}

# Generate the full path you would wget
fullpath=$url"distro=ubuntu,label=docker-builder,target=witherspoon/artifact/$deploy/images/witherspoon/"$filename

# wget the $fullpath, flash image on your system, and run tests

Debugging failures

In some cases, downstream hardware information (like network info and such) can not be shared upstream so it may be easiest to just have someone who has access to the downstream hardware and jenkins to manually update the gerrit commit with the relevant failure information. If this is not an issue, then debug information like the journal and what test failed could be uploaded as a gist to github or some other cloud storage and linked in the gerrit failure message.