Difference between revisions of "Automatic Builds of GlueX Software"

From GlueXWiki
Jump to: navigation, search
(Nightly Build)
(18 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
== Nightly Build ==
 +
 
Every night a complete build of the source directory is done on several platforms at the lab.
 
Every night a complete build of the source directory is done on several platforms at the lab.
  
* The builds are located in the directory /u/scratch/gluex/nightly on the JLab CUE. Everyday a new directory, named by date, is created in this directory, for example, /u/scratch/gluex/nightly/2014-10-03. In the daily directory there is a separate directory created for each platform, e. g., Linux_RHEL7-x86_64-gcc4.8.3.
+
* The builds are located in the directory /u/scratch/gluex/nightly on the JLab CUE. Everyday a new directory, named by date, is created in this directory, for example, /u/scratch/gluex/nightly/2020-09-23. In turn, in this directory, there is a separate directory created for each platform, e. g., Linux_RHEL7-x86_64-gcc4.8.5.
* Since the scratch disk cleaning job deletes files unread for more than two weeks, builds older than that cannot be used.
+
* Each platform-specific directory contains a copy of the version set file used in the build. There is also a "version.xml" in the same directory that is a soft link to this version set file. For example:
* The HDDS, and sim-recon packages are built. Latest versions of each are used.
+
/u/scratch/gluex/nightly/2020-09-23/Linux_RHEL7-x86_64-gcc4.8.5/version.xml
* The script run is /home/gluex/bin/nightly.sh. It is scheduled as a cron job for the [[GlueX shared account on the JLab CUE|"gluex" account]] on jlabl1.jlab.org. The job runs at midnight daily.
+
is a soft link to
* One cron job on jlabl1 runs the builds on the various platforms, as username gluex, serially. The current platforms are:
+
/u/scratch/gluex/nightly/2020-09-23/Linux_RHEL7-x86_64-gcc4.8.5/version_2020-09-23.xml
** jlabl1.jlab.org (RedHat Enterprise Linux 6, i686)
+
* Since the scratch disk cleaning job deletes files unread for more than 60 days, builds older than that are usually deleted.
** jlabl3.jlab.org (RedHat Enterprise Linux 6, x86_64)
+
* The hdds, halld_recon, halld_sim, hdgeant4, and gluex_root_analysis packages are built. The master branch of each are used.
** ifarm1101.jlab.org (CentOS 6.2, x86_64)
+
* The script run is /home/gluex/bin/nightly.sh. It is scheduled as a cron job for the [[GlueX shared account on the JLab CUE|"gluex" account]] on sandd1.jlab.org. The job runs at midnight daily.
** ifarm1401.jlab.org (CentOS 6.5, x86_64)
+
* Tne cron job on runs the builds on the various platforms, as username gluex, in parallel. The current platforms are:
** lorentz.jlab.org (RedHat Enterprise Linux 7, x86_64)
+
** jlabl5.jlab.org (RedHat Enterprise Linux 7, x86_64)
* Log files of the builds are created in the daily directory, for example, /u/scratch/gluex/nightly/2013-10-03/halld_jlabl3.log . For a particular platform, the output of both non-debug and debug builds are in a single log file.
+
** ifarm1901.jlab.org (CentOS 7.7, x86_64)
 +
* Log files of the builds are created in the daily directory, for example, /u/scratch/gluex/nightly/2020-09-23/halld_jlabl5.log .
 
* A summary of errors and warnings from the last build is available at https://halldweb.jlab.org/nightly/nightly_build_errors.txt . This file is only updated for builds that have errors or warnings. For clean builds, no summary log is produced.
 
* A summary of errors and warnings from the last build is available at https://halldweb.jlab.org/nightly/nightly_build_errors.txt . This file is only updated for builds that have errors or warnings. For clean builds, no summary log is produced.
 
** The last seven logs produced are archived in the same web directory:  https://halldweb.jlab.org/nightly/
 
** The last seven logs produced are archived in the same web directory:  https://halldweb.jlab.org/nightly/
 
* The summary of errors and warnings is also sent to the "nightly_build" [[Simple Email Lists|simple email list]].
 
* The summary of errors and warnings is also sent to the "nightly_build" [[Simple Email Lists|simple email list]].
* To use one of the nightly builds, you can set up the environment as follows (assume you want to use August 31, 2015):
+
* To use one of the nightly builds, you can set up the environment as follows (assuming you want to use the build from September 23, 2020):
 +
gxenv /u/scratch/gluex/nightly/2020-09-23/Linux_RHEL7-x86_64-gcc4.8.5/version.xml
 +
The [https://halldweb.jlab.org/docs/build_scripts_web/node6.html#SECTION00062400000000000000 gxenv command] is described in the [https://halldweb.jlab.org/docs/build_scripts_web/ Build Scripts document]. An alternate pair of environment setting command is
 +
source /group/halld/Software/build_scripts/gluex_env_nightly.sh 2020-09-23
 +
for bash and
 +
source /group/halld/Software/build_scripts/gluex_env_nightly.csh 2020-09-23
 +
for tcsh.
 +
 
 +
===Note on ssh scheme===
 +
 
 +
As mentioned above although the cron job runs on jlabl1, the builds are all actually done on other nodes. To do this without having to supply a passphrase, the cron job uses a special ssh private/public key pair that only allows the target script on the remote node (and no other command) to run only if the ssh connection comes from jlabl1 and if the target account holds the appropriate public key. This key has no passphrase associated with it<ref>If it did, then that passphrase would have to somehow be incorporated into scripts, a practice which is generally discouraged for security reasons.</ref>and thus can be used from a cron job. The remote target script is only mentioned in the authorized_keys file of the remote account. Only the ssh invocation is seen in the script (/home/gluex/bin/nightly.sh) on the local host (jlabl1).
 +
 
 +
Note that this special key pair is not the one used for standard ssh connections to gluex account on the CUE. The standard pair has a passphrase. This passphrase-less technique is described in a [http://www.linuxjournal.com/article/8257 2005 Linux Journal article] and a [http://cybermashup.com/2013/05/14/restrict-ssh-logins-to-a-single-command/ CyberSmashup blog article].
 +
 
 +
==Github Pull Request Tests==
 +
 
 +
Whenever a pull request for the halld_recon or halld_sim Github repositories is created, a test build is generated at JLab. Test builds are also generated if on open pull request has new commits to its associated branch.
  
  source /group/halld/Software/build_scripts/gluex_env_nightly.csh 2015-08-31
+
* The notification from Github is passed along via the "webhook" functionality. This particular webhook is configured to send a HTTP POST message to halldweb.jlab.org/cgi-bin/build/build_sim-recon_wehook.py . The web server halldweb is configured to allow python scripts to run in that directory and requires a user/password combination specific to that directory, using simple HTTP authentication.
 +
* The python cgi-script looks for the relevant pull request event, and starts a test build on the sandd1 VM using the same ssh scheme as the nightly builds described above.
 +
* These test builds use the software stored in /group/halld/Software/build_scripts/pull_request
 +
* The ssh command from the python script on halldweb calls build_pull_request_service_sshwrapper.sh. The procedure is managed by build_pull_request_service.sh, while the actual test build is performed in build_pull_request.sh.
 +
* The test builds are stored in /work/halld/pull_request_test. Debug logs are stored in this directory as ssh_log^REPO^BRANCH and env^BRANCH where "REPO' is "halld_recon" or "halld_sim" as appropriate and "BRANCH" is the name of the branch associated with the pull request. The test builds and build logs are stored in the directory REPO^BRANCH.
 +
* Several simple run-time tests with simulation and data are performed in test_pull_request.sh
 +
* The results of the test are posted as a comment on the pull request using leave_pull_request_comment.py, which sends a simple command to the Github REST API over HTTPS using the python requests library (note that this script uses python3).
  
=Note on ssh scheme=
+
==Nightly Code Analysis (scan-build)==
 +
The [http://clang-analyzer.llvm.org scan-build] analyzer that uses features built into the [http://clang.llvm.org clang] compiler is run nightly to produce a web report on potential problems identified in the code. Here's some details:
  
As mentioned above although the cron job runs on jlabl1, the builds are all actually done on other nodes. To do this without having to supply a passphrase the cron job uses a special ssh private/public key pair that only allows the target script on the remote node (and no other command) to run only if the ssh connection comes from jlabl1 and if the target account holds the appropriate public key. This key has no passphrase associated with it<ref>If it did, then that passphrase would have to somehow be incorporated into scripts, a practice which is generally discouraged for security reasons.</ref>and thus can be used from a cron job. The remote target script is only mentioned in the authorized_keys file of the remote account. Only the ssh invocation is seen in the script (/home/gluex/bin/nightly.sh) on the local host (jlabl1).
+
* Results of the latest analysis can always be found here: [https://halldweb.jlab.org/scan-build/LATEST https://halldweb.jlab.org/scan-build/LATEST]
 +
* The scan-build program is run via cron job using the ''gluex'' account and is run only on ifarm1401
 +
* The cron job runs the script ''/home/gluex/bin/nightly-scan-build.csh'' which is maintained in svn here: [https://halldsvn.jlab.org/repos/trunk/home/gluex/bin https://halldsvn.jlab.org/repos/trunk/home/gluex/bin]
 +
* This piggybacks from the nightly build of sim-recon located in ''/u/scratch/gluex/nightly/''. It requires the setenv.csh script exists for the gcc compiler so that it can source it to get basic things like ROOT, XERCES, JANA, CCDB set up in the environment. (Could be changed to use gluex_env instead and may be done in the future.)
 +
* This requires the clang compiler. It is currently hardwired to use the 3.7.0 compiler installed in ''/group/halld/Software/ExternalPackages/clang-llvm/llvm_clang_3.7.0''
  
Note that this special key pair is not the one used for standard ssh connections to gluex account on the CUE. The standard pair has a passphrase. This passphrase-less technique is described in a [http://www.linuxjournal.com/article/8257 2005 Linux Journal article].
 
  
 
----
 
----
  
 
<references/>
 
<references/>

Revision as of 15:44, 23 September 2020

Nightly Build

Every night a complete build of the source directory is done on several platforms at the lab.

  • The builds are located in the directory /u/scratch/gluex/nightly on the JLab CUE. Everyday a new directory, named by date, is created in this directory, for example, /u/scratch/gluex/nightly/2020-09-23. In turn, in this directory, there is a separate directory created for each platform, e. g., Linux_RHEL7-x86_64-gcc4.8.5.
  • Each platform-specific directory contains a copy of the version set file used in the build. There is also a "version.xml" in the same directory that is a soft link to this version set file. For example:
/u/scratch/gluex/nightly/2020-09-23/Linux_RHEL7-x86_64-gcc4.8.5/version.xml

is a soft link to

/u/scratch/gluex/nightly/2020-09-23/Linux_RHEL7-x86_64-gcc4.8.5/version_2020-09-23.xml
  • Since the scratch disk cleaning job deletes files unread for more than 60 days, builds older than that are usually deleted.
  • The hdds, halld_recon, halld_sim, hdgeant4, and gluex_root_analysis packages are built. The master branch of each are used.
  • The script run is /home/gluex/bin/nightly.sh. It is scheduled as a cron job for the "gluex" account on sandd1.jlab.org. The job runs at midnight daily.
  • Tne cron job on runs the builds on the various platforms, as username gluex, in parallel. The current platforms are:
    • jlabl5.jlab.org (RedHat Enterprise Linux 7, x86_64)
    • ifarm1901.jlab.org (CentOS 7.7, x86_64)
  • Log files of the builds are created in the daily directory, for example, /u/scratch/gluex/nightly/2020-09-23/halld_jlabl5.log .
  • A summary of errors and warnings from the last build is available at https://halldweb.jlab.org/nightly/nightly_build_errors.txt . This file is only updated for builds that have errors or warnings. For clean builds, no summary log is produced.
  • The summary of errors and warnings is also sent to the "nightly_build" simple email list.
  • To use one of the nightly builds, you can set up the environment as follows (assuming you want to use the build from September 23, 2020):
gxenv /u/scratch/gluex/nightly/2020-09-23/Linux_RHEL7-x86_64-gcc4.8.5/version.xml

The gxenv command is described in the Build Scripts document. An alternate pair of environment setting command is

source /group/halld/Software/build_scripts/gluex_env_nightly.sh 2020-09-23

for bash and

source /group/halld/Software/build_scripts/gluex_env_nightly.csh 2020-09-23

for tcsh.

Note on ssh scheme

As mentioned above although the cron job runs on jlabl1, the builds are all actually done on other nodes. To do this without having to supply a passphrase, the cron job uses a special ssh private/public key pair that only allows the target script on the remote node (and no other command) to run only if the ssh connection comes from jlabl1 and if the target account holds the appropriate public key. This key has no passphrase associated with it[1]and thus can be used from a cron job. The remote target script is only mentioned in the authorized_keys file of the remote account. Only the ssh invocation is seen in the script (/home/gluex/bin/nightly.sh) on the local host (jlabl1).

Note that this special key pair is not the one used for standard ssh connections to gluex account on the CUE. The standard pair has a passphrase. This passphrase-less technique is described in a 2005 Linux Journal article and a CyberSmashup blog article.

Github Pull Request Tests

Whenever a pull request for the halld_recon or halld_sim Github repositories is created, a test build is generated at JLab. Test builds are also generated if on open pull request has new commits to its associated branch.

  • The notification from Github is passed along via the "webhook" functionality. This particular webhook is configured to send a HTTP POST message to halldweb.jlab.org/cgi-bin/build/build_sim-recon_wehook.py . The web server halldweb is configured to allow python scripts to run in that directory and requires a user/password combination specific to that directory, using simple HTTP authentication.
  • The python cgi-script looks for the relevant pull request event, and starts a test build on the sandd1 VM using the same ssh scheme as the nightly builds described above.
  • These test builds use the software stored in /group/halld/Software/build_scripts/pull_request
  • The ssh command from the python script on halldweb calls build_pull_request_service_sshwrapper.sh. The procedure is managed by build_pull_request_service.sh, while the actual test build is performed in build_pull_request.sh.
  • The test builds are stored in /work/halld/pull_request_test. Debug logs are stored in this directory as ssh_log^REPO^BRANCH and env^BRANCH where "REPO' is "halld_recon" or "halld_sim" as appropriate and "BRANCH" is the name of the branch associated with the pull request. The test builds and build logs are stored in the directory REPO^BRANCH.
  • Several simple run-time tests with simulation and data are performed in test_pull_request.sh
  • The results of the test are posted as a comment on the pull request using leave_pull_request_comment.py, which sends a simple command to the Github REST API over HTTPS using the python requests library (note that this script uses python3).

Nightly Code Analysis (scan-build)

The scan-build analyzer that uses features built into the clang compiler is run nightly to produce a web report on potential problems identified in the code. Here's some details:

  • Results of the latest analysis can always be found here: https://halldweb.jlab.org/scan-build/LATEST
  • The scan-build program is run via cron job using the gluex account and is run only on ifarm1401
  • The cron job runs the script /home/gluex/bin/nightly-scan-build.csh which is maintained in svn here: https://halldsvn.jlab.org/repos/trunk/home/gluex/bin
  • This piggybacks from the nightly build of sim-recon located in /u/scratch/gluex/nightly/. It requires the setenv.csh script exists for the gcc compiler so that it can source it to get basic things like ROOT, XERCES, JANA, CCDB set up in the environment. (Could be changed to use gluex_env instead and may be done in the future.)
  • This requires the clang compiler. It is currently hardwired to use the 3.7.0 compiler installed in /group/halld/Software/ExternalPackages/clang-llvm/llvm_clang_3.7.0



  1. If it did, then that passphrase would have to somehow be incorporated into scripts, a practice which is generally discouraged for security reasons.