Building DAQ-PROD on the SPLTS Cluster


This page describes how to checkout, build and deploy the DAQ-PROD meta-project onto the SPLTS cluster. You will be able to view deployed MBeans via their JMX consoles. A tidy Table of Consoles is available for you to reach the nodes directly. Your SSH Tunnels must be setup before hand.

  1. Announce the upcoming deployment
  2. Setup your SSH tunnels
  3. Create a workspace for DAQ-PROD
  4. Checkout and build DAQ-PROD
  5. Undeploying the current DAQ-PROD
  6. Initialize the Postgres DB
  7. Load the Steering Files
  8. Load the Postgres DB
  9. Deploying the new DAQ-PROD
  1. Announce the upcoming deployment

    2. Since the SPLTS cluster is designed for development and testing, the odds are good that someone else is mucking around with DAQ compononents. It has been decided that whenever something major is planned for the SPLTS cluster, it be announced on the SPLTS Skype chat room.
    ad Contact     Martin Stoufer and he will invite you into the room.

    A simple note telling other users of your intentions and time frame for deployment will suffice. It is understood that during this time, the cluster is "Hands Off" to other users and any jboss-centric processes will be stopped as needed.

  2. Setting up your SSH tunnels into the cluster nodes

    3. To make debugging and monitoring easier, you need to start multiple terminal windows on your local workstation, one per SSH connection. You may automate this step as needed. One ssh command per new terminal window:

      $ ssh splts-access
      $ ssh splts-dbs
      $ ssh splts-evbuilder
      $ ssh splts-ichubXX (all hubs)
      $ ssh splts-stringprocXX (all stringprocs)
      $ ssh splts-expcont

    You will notice that when you logon to the access node, an ssh-agent is started for you. This will facilitate the deployment of files to the remote nodes.

  3. Create a workspace for DAQ-PROD

    4. Create a new dir for the DAQ-PROD workspace (so it doesn't conflict with other bfd/cvs projects you may be working on)

      jboss@sptls-access[icecube] mkdir DAQ-PROD_WEASEL-00-01_ws
      jboss@sptls-access[icecube] cd DAQ-PROD_WEASEL-00-01_ws

  4. Checkout and build DAQ-PROD
    5.   jboss@sptls-access[DAQ-PROD_WEASEL-00-01_ws] bfd co -r VWEASEL-00-01 DAQ-PROD
      jboss@sptls-access[DAQ-PROD_WEASEL-00-01_ws] bfd init $ICECUBE_TOOLS
      jboss@sptls-access[DAQ-PROD_WEASEL-00-01_ws] source setup.sh
      jboss@sptls-access[DAQ-PROD_WEASEL-00-01_ws] ant -q clean.ws lib.all

    The order here may look a bit backwards, but this allows the intended build state of DAQ-PROD to be preserved in new BFD workspaces.

  5. Undeploying the current DAQ-PROD

    6. Before a deployment can occur, we must bring all the nodes to a common starting execution point. This involves stopping jboss on each node, removing old DAQ-PROD deployments, and deploying the new version.

    1. On the splts-expcont node, you can invoke the anvil command and remotely shutdown the running jboss process on all the other nodes. Once you 'su' to the jboss user:
      2.   jboss@sptls-expcont[icecube] anvil
        Anvil@splts-expcont** nojboss
        Anvil@splts-expcont** exit

      This will take 20 seconds on each node. Any critical errors will be displayed to the terminal window. Shutting down a jboss server that was not running is OK.

    2. Once the servers are stopped, you can now remove the EAR files from each node. DAQ-PROD provides a smart little script to do this for you. Back on the splts-access node:
      3.   jboss@sptls-access[DAQ-PROD_WEASEL-00-01_ws] cd cluster-config/bin
        jboss@sptls-access[bin] ./deploy-daq.sh -utq
        jboss@sptls-access[bin] ./deploy-daq.sh -utqe

    3. Once this completes, you can start jboss servers and deploy the new EAR files. It is highly advisable that you start them in the following order. Wait until jboss starts completely on splts-dbs before continuing to the other nodes. Again, on the expcont node:
      4.   jboss@sptls-expcont[bin] anvil
        Anvil@splts-expcont** jboss
        Anvil@splts-expcont** exit

  6. Initialize the Postgres DB

    7. With JBoss running (so the Postgres DB is visible to JBoss) run the daq-db-init.sh program.

      jboss@sptls-access[bin] cd ../../daq-db-init
      jboss@sptls-access[daq-db-init] ./daq_db_init.sh

  7. Load the Steering Files

    8. This is necessary after running daq-db-init. From the top of the new workspace:

      jboss@sptls-access[daq-db-init] cd ../domhub-prod-conf/resources
      jboss@splts-access[resources] ./load_steering.sh -t
    This will print a few error messages about duplicate primary keys which are currently safe to ignore

  8. Load the Postgres DB

    9.   jboss@sptls-access[SPLTS] cd ../../../cluster-config/bin/
      jboss@sptls-access[bin] ./componentsInConfig.sh

    If you see SQLExceptions, try running the command again, it should run without any errors.

    Strictly speaking, this is only needed if one of the *.sql files have been modified since the last time you initialized the DB. It doesn't hurt to re-initialize it, so feel free to do it if you have any doubt.
  9. Deploying the new DAQ-PROD

    10. Now deploy the new DAQ-PROD from splts-access.

      jboss@sptls-access[bin] ./deploy-daq.sh -btq
      jboss@sptls-access[bin] ./deploy-daq.sh -dtqe
      jboss@sptls-access[bin] ./deploy-daq.sh -dtq

    At this point you have now deployed a new DAQ-PROD onto the SPLTS cluster. You should see deployment info scroll by on each terminal window for the cluster nodes. Any errors should be noted and passed onto the responsible developer. Being an announced DAQ-PROD release, the amount of errors should be either ignorable, or non-existent!

  10. Start the IceAxe Portal

    11. Assuming the anvil commands worked above, the jboss server is running on the expcont node. The Portal should be running and available at http://localhost:14200/IceAxe. If you cannot reach it, please refer to the IceAxe documentation.

This page maintained by Martin Stoufer
Page last modified: