September 26, 2001

History and Status Report on "Test-Board Quartet"/"String 18 Upgrade" Communications

Since the time of the string 18 installation at the pole,  the communication with DOMs was carried out through a 'terminal server' connected to serial ports on  'DESY DAQ' cards in a CPCI card cage in the left most rack of the AMANDA counting house at the south pole.

The communications speed, 38 Kilobaud per second, is the fastest serial communications data rate for the DOM because both the UART in the DOM CPU and the UART in the terminal server have it in common.  DOM bound data passed, unbuffered, from the terminal server to the test-board  RS-232 to CMOS converter, to the  modulator circuitry in the test-board, and out over the twisted pair to the DOM.  38 Kilobaud ASCII serial communications will remain the fall-back communications mode to the DOMs, available at boot time for diagnostics and recovery.  Once booted, the DOMs in IceCube will be shifted to a faster data rate, and a protocols better suited to data acquisition mode.

Last January a small crew from  LBNL installed a quartet of test-boards in another chassis in the rack at the pole as seen in the photo above.

The chassis contained a PC-104 form-factor CPU.  All four test-boards interconnect with  the PC-104 system bus.  This intimate connection allowed timing measurements to be made when 38 Kbaud serial communications is temporarily suspended. I.e. the server CPU tells the DOM to switch to timing event mode, the PC-104 CPU takes control of the communications modulator in the test-board to generate a timing tick, the DOM receives the tick, and sends an answering tick which is received in the test-board FPGA.  At that point the test-board data, two time-stamps and a waveform are read out, and the DOM told to forward its timing event data, also two time-stamps and a waveform.  A timing event would take about one or two seconds due to the high overhead. The data reduces to two equations in two unknowns: the electrical length of the cable, and the time offset between the clock in the test board and the clock in the DOM.

It was recognized over a year ago that having the four test-boards on the PC-104 system bus would allow faster communications, and the interleaving of communications with time-ticks.  Fiscal constraints and priorities caused the development of PC-104 CPU based communications to be stretched out over the past eight months. Since (slow) communications was in place, the effort focused on proving that time synchronization could meet IceCube requirements. The same old serial communications through the terminal server has served  in the lab here at LBNL and at the pole all summer long.

Due to the efforts of John Jacobsen working on Linux system software, driver software, and data acquisition code,  and Gerald Przybylski working on test-board firmware, the milestone of communicating between the server CPU and a DOM through the PC-104 CPU  is achieved.  The terminal server is, thus, made redundant. (This new code can be installed at pole at any time.  The terminal server capability is still maintained as a back-up.)
 
 

This is how the new communications scheme works:    Messages from the server CPU destined for the DOM pass from the server CPU through a TCP/IP socket to the PC-104 CPU, where a daemon writes them as bytes to the transmit FIFO in the FPGA.  Firmware in the FPGA converts bytes into serial data for transmission to the DOM over the twisted quad cable.  Messages from the DOM are detected (by a digital discriminator with automatic recentering to follow input level variations and low frequency noise), and fed to a transition decoder which turns the signal back into serial RS-232 data.  The serial signal drives the CMOS to RS-232 chip on the test-board which drives the terminal server. The serial data also drives the UART deserializer in the FPGA.  The deserializer writes bytes into the receive FIFO.  When the FIFO is nearly full, or when there are some bytes present and no more received for several character periods, the FIFO wrapper circuitry in the FPGA asserts an interrupt to the PC-104 CPU which activates driver code to transfer data to CPU memory. The dataserver daemon passes the bytes back to the remote server CPU through a TCP/IP socket.  Care was taken that the socket interface to the terminal server behaved identically to the socket interface to the dataserver daemon on  the PC-104 CPU. No other data acquisition software needed to be rewritten, and all the functionality has been retained.

Having the PC-104 CPU perform both the data communication and timing events, in combination with a pre-emption capability in the FPGA, a complete time calibration event to be reduced to a few milliseconds.  This goal will be within reach when communications in the DOM migrates to a UART in the FPGA.

The string 18 upgrade scheduled for the December '01, January '02 south pole summer  will replace the original DESY data acquisition boards with DESY revised test-boards on passive back planes using a single board CPU's bus.   The software running on our PC-104 CPU will be ported to the CPU specified by DESY (Karl-Heinz Sulanke).   The firmware for the LBL test-board FPGA will be ported to the DESY version of the test-board during the October system integration.  A first working, configured system will be integrated. here at LBNL   That system will be cloned into five systems which will be shipped to the south pole for installation in December of 2001.  A system will be retained at  LBNL to assist in debugging and development.

The upcoming installation provides us with an opportunity to attempt remote commissioning of hardware at the pole.  The idea is that software and firmware are developed on systems in the northern hemisphere which are functionally identical to systems installed at the south pole.  When the satellite is 'up', providing communications, the pre-tested software and firmware are downloaded to the hardware at the pole. Most of the time it is expected that no remote or local debugging will be required to make the system at the pole function in the way  the development system functioned in northern latitudes.

The glossary (as if you need one...  ;-)):
CPCI Compact PCI form factor based data acquisition card cage based on Intel/AMD x86 technology and the PCI bus.
DESY Deutsches Elektronen-Synchrotron (DESY Zeuthen)
DOM Digital Optical Module which contains a photomultiplier tube, high speed signal digitizing electronics and communications electronics for data recovery.
FIFO First-In First-Out Memory
FPGA Field Programmable Gate Array which is configured after application of power.
PC-104 Compact Single Board Computer for embedded systems based in Intel/AMD x86 technology which interfaces with peripherals through the PC-104 bus which is related to the ISA bus.
TCP/IP Transmission Control Protocol/Internet Protocol
UART Universal Asynchronous Receiver Transmitter - logic block or hardware device for converting serial RS-232 protocol data to or from bytes one-at-a-time.

Questions and comments may be directed to 

Sep. 28, 2001 09:46