Preliminary specifications for the IceCube DOM Hub.

May 15, 2001, Gerald Przybylski
 
Number of channels per DOM hub board tentatively 20 per board, but there are arguments for 60 per board
DOM Hub chassis shall have line power supply for each DOM Hub internal to the unit itself
DOM Hub chassis shall have line power supply for all served DOMs internal to the unit itself (assume about 2.5 W per DOM)
DOM Hub shall have a 10 bit 40 MHz ADC on each twisted pair
DOM Hub shall have an 8 bit DAC on each twisted pair
DOM Hub shall have on board oscillator for stand-alone operation
DOM Hub shall have external reference oscillator input with transformer coupling and common mode choke.
DOM Hub shall have external 1 PPS reference input for time synchronization.  Transformer and common mode choke shall be used.
Each twisted pair shall have provisions for interrupting the power to allow selective rebooting of the DOMs on it.
Each twisted pair shall have current monitoring functionality built into it. Current monitoring should be DC-isolated
The FPGA shall be wired to external fast static RAM memory
The FPGA shall be wired to a footprint for a daughter card ARM CPU (e.g. the Brightstar Engineering NanoEngine) (note below)
The daughter card CPU shall have ethernet capability
The FPGA shall have wiring to a footprint for ethernet magnetics.
The FPGA shall be pinned out to accommodate substitution of an FPGA with a hard core ARM processor built in. (note below)
The DOM Hub FPGA shall be loadable via JTAG for testing and maintenance
The DOM Hub PLD shall be loadable via JTAG.
The DOM Hub chassis shall be a 1U (if serving 20 DOMS) or at most 2 U (if serving 60 DOMs) rack mountable unit 
DOM Hubs shall plug into their individual rack locations. Plugging into power, network, and DOM cables will not require service personnel to go from one side of the rack to the other to remove a unit. Connections will be in the back of the rack.
The DOM Hub front panel shall have a power indicator light, a status light activated by the FPGA, and perhaps, a network activity status light, and perhaps an ethernet link light.  These lights will be on the front panel.  Lights will be of LED type.
Provision will be made for distribution of house cooling air, or preferably, temperature controlled fans to enforce a level of thermal regulation.  (systems operated at constant temperature are more reliable).
Low power parts will be used wherever possible
High efficiency power supplies will be used wherever possible.

We will assume it will be feasible to operate the DOM Hub with an Altera Excalibur chip with an ARM CPU sharing the same chip as the FPGA.  With the CPU in the same package as the FPGA, the system interfaces would be greatly simplified.  The cost of the daughter card CPU could also be avoided.  When resources become available, this scheme will have to be evaluated.  In the absence of an evaluation, the proven technology of FPGA as programmable peripheral controller will be used.

Additional design requirement suggestions are invited. Please email 

from a previous version:

Front End Electronics in the counting house; The DOM Hub
 

Requirements

1.  Deliver power to DOMs through the same twisted pair used for communications.

2.  Transfer communications data bidirectionally to the  DOM.

3.  Transfer communications data bidirectionally to computers on a Local Area Network (LAN).

4.  Exchange timestamped time-ticks with the DOM.

5.  Confine special hardware and protocols as close to the DOM twisted pair cable as possible.

6.  For maintenance, provide transparent bidirectional communications between DOMs and computers on the LAN.

7.  Minimize power consumption, and heat load.

8.  Convenient packaging for the counting house or for use as a test stand during  production or for R&D at participating institutions.

9.  Consist, as much as possible of commodity components and assemblies.  Minimize specialized boards.

10. Design to exploit economies of scale.

11. Provide a high degree of flexibility and configurability.

12. Run the Linux operating system to avoid license fees for operating systems. Embedded Linux preferred.

13. Minimize I/O overhead for short to medium packet length.

14. Aggregate a reasonable number of DOM channels into one data stream.

15. Synchronize timing pulses to a master 10 MHz clock, a 1 pulse per second clock tick and a once per second time string.

16. Be highly reliable in a lights-out environment.

Packaging

DOM channels per board should be supported in multiples of four.  As of February 2001, the plan calls for each pair of wires in the cable to support two DOMs.  Each pair in the cable is half of a quad cable. Quad cables terminate in a four pin connector.  Hence the requirement of multiples of four.

The packaging should be mountable in a 19 inch wide x 42 U high relay rack.

The electronics house may provide forced air for cooling, or the packaging may provide an internal fan for cooling. Fan units should be provided for cooling if not part of a card cage system, if a card cage style packing is chosen.

If built into 1U enclosures, the enclosure nay contain an internal fan, or a rear panel tube for interconnection with a plenum or manifold distributing cooling air.  If the 1U package contains an internal fan, then fan failure or over temperature condition should be detected.

Power consumption and efficiency

The high cost of fuel delived to the pole, and the limited power capacity of the generators at the station motivate the concern for system efficiency.  Therefore, components on the circuit board should be chosen with attention to low power consumption.  Power supply components, packaged power supplies, on-board power supply designs and components should be chosen with attention to low power consumption and high efficiency.

Power Delivery to DOMs

The DC power delivered to the twisted pair by the DOM Hub must pass through a low pass filter which isolates twisted pairs from each other and from the power supply  in the communications band by of order 60 dBV.
The filter should also minimize the power loss through it arising from resistive elements in the current path as much as possible.  The filter components should occupy a minimum of PC board real estate.

Communications and timing signals

Communications signals from the DOMs will be received by an ADC, one per DOM
Compensation for cable frequency roll-off charactiristics (transfer function) will be done by digital filter implemented in firmware on the DOM Hub board.
Show speed communicatioons for start-up and maintenance shall be supported, as well as high speed communications.
High speed communicatins refers to baud rates in excess of 100 kbps (target 200 kbps, or perhaps higher).

Communications signals sent to the DOM will be through a DAC, one per DOM.  The current candidate (implemented on the '99-'00 DOM) is the AD ???? current DAC which features transformer output compatibility.

The chassis will provide an input for distributed 10 MHz frequency standard and  one pulse per second time tick. The system computer may synchronize its local time with a time server through the Network Time Protocol (NTP).

Computing Environment

The counting house could be built around a VME or CPCI card cage infrastructure.  That infrastructure would consist of card cages mating to standard power supplies and rack fan units.  The comuting resources would then be supplied by standard CPUs which plug into a system backplane in the card cage.  The card cages would be populated with additional custom front-end cards which communicate by system backplane to the adjacent CPU, and in the usual manner to the DOM twisted pair cables.  Power for the DOMs can be distributed from a suitable common power supply in the electronics house or converted from power available from the crate power supply by the custom board.

VME/CPCI model advantages:
Standard off-the-shelf products
Factory support for CPUs
Support for many operating systems
String processor functionality could run in the card cage CPU
Easy upgradability of CPU resources.

The VME/CPCI model disadvantages:
VME and CPCI card cages and power supplies are expensive (calculated on a per DOM supported basis).
Standard card cage power supplies have capacities ill suited to the power needs of the DOM hub cards.
Relatively low density packing in a rack.
The CPU cards cost $3000 to $4000.  '
The CPUs consume of order 50 Watts, a great deal of which goes into the CPU, and bus interface chips.
The custom cards must also include bus interface chips which also consume power.
The model requires that if more than one CPU per card cage is supported, then the backplane must be segmented in a manner that supports the number of subsystems in the cardcage.
High CPU power consumption (of order 30 to 50 Watt per CPU (approximately 100 mW/MHz) card amortized across one to four DOM hub cards)
Network bandwidth amortized over one to four DOM hub cards.
Custom driver required.  Memory mapped access to peripherals precluded.
Hard disk probably required for each CPU, or an NFS mounted disk from a server.
Could be very costly if it is found that one CPU per DOM Hub board is required. Furthermore, replacement backplanes would be required for the segmented card cages, or the electronics house would contain unused slots in the card cages.
Development of multi-slot high speed bus cards likely to be time consuming.

An alternate model is currently preferred consists of 1U packaging for the DOM Hub card, which has an embeddeed CPU daughter card mounted on it to provide computing resources, one per DOM Hub card.  The 1U chassis hardware containing power supplie deliver 5V and 3.3V to a DOM hub card.  The back panel of the chassis contains sockets for four or five quad cables if each quad supports four DOMs, or eight or 10 quad cables if each quad supports only two DOMs.  The mbedded CPU card will have to be a compact, low power product, like the BrightStar Engineering nanoEngine. The nanoEngine is a sub credit card size 200 MHz StrongARM CPU with built-in 10/100-Base-T ethernet.   As with the bus based model, the DOM Hub card would contain several FPGAs which comprize a configurable peripheral controller communicating with the CPU memory and address bus, as well as with the DAC and ADC chips.

Embedded CPU model advantages:
Inexpensive packaging (1U), widely available at highly compettitive prices, including power supply.
High packing density in a rack. Little wasted space.
Low CPU power consumption  (of order 2 Watts per 200 MHz CPU or 10 mW/MHz) amortized across only one card.
Inexpensive CPU card (200 MHz CPU, 32 MB Memeory, 4 MB Flash for $495 in 100 unit quantities)
100-Base-T network bandwidh dedicated to only one DOM Hub card.
Memory mapped access to DOM Hub FPGAs supported by the OS and the hardware.
Boots from flash.  TFTP or BOOTP boot.  DHCP supported.  No hard disk required.
Avoids backplane bus interface chips and the power they consume.
Avoids long high speed bus lines with multiple taps.

Embedded CPU model disadvantages:
Potential obsolescence of CPU board. -- (BrightStar Sales promises availability of the nanoEngine for at least 5 years)

Network environment

The DOM Hub will communicate with computers on the Local Area Net through a 100/100-Base-T interface.

The DOM Hub will support TCP/IP socket based communications  for each DOM to deliver physics hit data to string trigger computers.

The DOM Hub will provide transparent web based communications with individual DOMs so that maintenance may be done without the need of the full data acquisition to be running.

Packaging
The proposed packaging of the DOM Hub is a 1U Rack Chassis (19" wide x 1 3/4" high x (depth to be determined)).  The chassis will plug into AC mains, and have connectors for DOM quads on the back of it.

A possible alternate packaging is a rack card cage.  If a card cage is used the backplane of the card cage will be a custom board distributing power and which will have DOM cables permanently plugged into it.