GPS Clock misbehavior

Symptoms

A number of fault conditions have been observed at various points in the hardware and software chain. They are tabulated below:


Symptom
Reporter
Root cause, or best guess
Resolution
a
Faulty string detected by DOR Driver
John
Jacobsen
a. Faulty decoding by UART
b. Faulty string from GPS clock
c. Noise triggering UART
Faulty UART decoding may be another artifact of timing violation.
To date, no faulty strings have been detected at the serial output of the GPS clock (4/8-4/22/05)
b
19,999,999 counts paired with 20,000,001 counts in the next second
(Seen on all cards simultaneously)
Claire
Pettersen
Symptom induced by incorrect timing relationship between ↑ transitions of 10 MHz  and ↑ transitions of the 1Hz (or 1pps) signal. Change GPS clock cabling per recommendation
c
Δt = 40,000,000
Claire
Pettersen
Skipped time cal from driver due to suppression of malformed string (see a above)
Software needs to deal with occasional omitted tcal
c1
Δt = 40,000,000 or higher multiple of 20M
GTP
a. PCI  fault causing lost packet
b. Driver error causing packet to be skipped
c. Operating system fault causing lost packet
d. Missing 1 pulse per second pulse
Seen several times before 4/18/05.
Periodic on 1-day scale
Not seen since 4/18/05 with variant (logging) driver.

d
Δt not a multiple of 20,000,000 due to insertion of a zero byte into the the DOR time string,
or an 8 bit shift in the DOR time data.  Least significant bits appear to be discarded
(Seen in the same second on two of four DOM Hubs)
In the 4 cases  observed (by DC)  this error the DOR timestamp appears to be shifted up one byte, and the new lower byte was '0x01'. Fix: ignore such stamps.
Claire
Pettersen
Dima
Chirkin
Symptom induced by incorrect timing relationship between ↑ transitions of 10 MHz  and ↑ transitions of the 1Hz (or 1pps) signal Change GPS clock cabling per recommendation
e
GPS Quality character not a " " (Space) but *is* one of the other valid characters (usually anyway). --DC
Dima
Chirkin
Normal behavior: Observed in the bldg 50 test lab. Assumed to be caused by sub-optimal antenna placement resulting in tracking of an insufficient set of satellites. Symptom of reacquisition of satellites. See pix...
Mount GPS antenna clear of obstructions and shadows per recommendations in manual
f
GPS clock changed from a good to a bad value.
Dima
Chirkin
DOR card GPS String Buffer not Flushed when starting a data run., or another symptom of  incorrect timing relationship between ↑ transitions of 10 MHz  and ↑ transitions of the 1Hz (or 1pps) signa Software fix:: ignore the new value.
g
GPS clock changed from a bad to a good value.  Dima
Chirkin
DOR card GPS String Buffer not Flushed when starting a data run, or another symptom of  incorrect timing relationship between ↑ transitions of 10 MHz  and ↑ transitions of the 1Hz (or 1pps) signa Fix: if the new value is repeated more than once, accept the new value.
Calibration with the good GPS structure therefore depends on having at
least 3 GPS last structures in a ring buffer before performing a calibration (which is done in fat-reader, but may not be possible with the current data collector's setup).
I've (DC) just sent the corrected calibration class to Mark, where I *could* (I think) correct this type of error except when the bad GPS is recorded in the first tcal cycle.
h
Observed valid (usable) GPS-DOR structures which referred to UTC times at least hour before the current run. This may need to be fixed if dor clock can be reset during the start of new runs. Dima
Chirkin
Stale DOR time structures not flushed before starting a new run.
Future software fix?

Action plan

Clair Pettersen and Arthur Jones will collect correlated time strings directly from the GPS clock, and DOR driver(s).  Arthur can do it for only one DOR. Claire can do it for the UW FAT DAQ.
There is no plan at this time to ask that this be done at the pole;  we will see what the first data tells us, then rethink.

Measure and record RS-232 serial port waveforms distributed to DOR cards.

Logging

Exploring clock to 1pps skew

Suggestions welcome.

Email or Phone Gerald Przybylski 510 486 7165

Additional information:
 --DC -You can see the GPS errors in the log files (at the bottom of the pages linked to the dates in the calendar at http://icecube.wisc.edu/~dima/work/IceCube/03/