Last time, I built a stratum 1 NTP server with a PPS signal from a GPS receiver, synchronizing my server’s clock to within 10 microseconds of UTC. However, NTP was designed to synchronize clocks within a few tens of milliseconds over the Internet, and I’d be lucky to achieve millisecond accuracy on a LAN. I mentioned that PTP was the alternative that could achieve accuracy in the sub-microsecond range. Well, this time I’ll be setting up PTP between my server and my PC with the hardware timestamping on the ConnectX-3s.
If you are following along at home, don’t despair if your hardware can’t do timestamping or PTP. I will also attempt to set up PTP with software timestamping later for my other devices.
Naturally, I first turned to the
gpsd documentation, since that was a decent
reference for setting up NTP with the PPS signal. Well, this is what it says for
PTP with hardware timestamping:
Sadly, theory and practice diverge here. I have never succeeded in making hardware timestamping work. I have successfully trashed my host system clock. Tread carefully. If you make progress please pass on some clue.
That didn’t sound encouraging at all. “Oh well, I guess I am on my own here,” I
thought to myself. “How bad could digging through a few
man pages and random
online documentation be? Worst case, there is the source code, right?”
The PTP implementation that everyone talks about is
linuxptp, so I started by
installing it on my server:
$ sudo apt install linuxptp Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: linuxptp 0 upgraded, 1 newly installed, 0 to remove and 34 not upgraded. Need to get 177 kB of archives. After this operation, 773 kB of additional disk space will be used. Get:1 http://deb.debian.org/debian bullseye/main amd64 linuxptp amd64 3.1-2.1 [177 kB] Fetched 177 kB in 0s (6,335 kB/s) Selecting previously unselected package linuxptp. (Reading database ... 506647 files and directories currently installed.) Preparing to unpack .../linuxptp_3.1-2.1_amd64.deb ... Unpacking linuxptp (3.1-2.1) ... Setting up linuxptp (3.1-2.1) ... Created symlink /etc/systemd/system/multi-user.target.wants/timemaster.service → /lib/systemd/system/timemaster.service. Processing triggers for man-db (2.9.4-2) ...
It installed this
timemaster service, which is a configuration generator to
make it possible to run NTP with PTP as a clock source. It seems like an
interesting tool, but my server would be the clock source for the entire network
(“grandmaster” in PTP terminology), so I definitely wouldn’t want it. It doesn’t
help that the default configuration file that comes with the
doesn’t seem to work and the Debian documentation is non-existent1, so let’s
just disable it:
$ sudo systemctl disable --now timemaster.service Removed /etc/systemd/system/multi-user.target.wants/timemaster.service.
Now, the actual underlying
linuxptp utilities are:
ptp4l, which is the actual PTP daemon. It announces “PTP time” (really TAI) in hardware timestamping mode and UTC in software timestamping mode; and
phc2sys, which synchronizes between the PTP hardware clock (PHC) and the system clock. When running
ptp4lin hardware timestamping mode, the hardware clock needs to be synchronized to the system clock.
Setting up hardware timestamping PTP
I decided to dive straight into the exciting bit, because why not? I started by
ptp4l between my server and PC on the ConnectX-3s.
Now, there are three possible transports for PTP:
- IEEE 802.3 network transport, which is a fancy way of saying raw Ethernet with
0x88F7. This is option
- UDP IPv4 network transport. This is option
- UDP IPv6 network transport. This is option
Note that for UDP transport, this typically is done over multicast.
I ended up choosing the IEEE 802.3 transport since I am using Linux bridges to link my ConnectX-3 to the rest of my Ethernet network and bridges don’t support hardware timestamping. I didn’t feel like fiddling with multicast on an interface that has no IP, so raw Ethernet it was. In a less janky setup with a real 40 GbE switch, I’d probably have used multicast UDP.
On the server, I ran (pretend
cx3 is the name for the ConnectX-3 interface,
-m is for printing to
stdout instead of
$ sudo ptp4l -2mi cx3 ptp4l[94881.798]: selected /dev/ptp0 as PTP clock ptp4l[94881.858]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[94881.858]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[94888.005]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES ptp4l[94888.005]: selected local clock [server EUI-64] as best master ptp4l[94888.005]: port 1: assuming the grand master role
On the client, I ran (
-s to mean a “slave” clock):
$ sudo ptp4l -2msi cx3 ptp4l[17633.604]: selected /dev/ptp1 as PTP clock ptp4l[17633.650]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[17633.650]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[17634.466]: port 1: new foreign master [server EUI-64]-1 ptp4l[17638.466]: selected best master clock [server EUI-64] ptp4l[17638.466]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[17640.466]: master offset 11088 s0 freq -20470 path delay 138 ptp4l[17641.466]: master offset 11286 s2 freq -20272 path delay 155 ptp4l[17641.466]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[17642.466]: master offset 11300 s2 freq -8972 path delay 155 ptp4l[17643.466]: master offset 29 s2 freq -16853 path delay 155 ptp4l[17644.466]: master offset -3360 s2 freq -20233 path delay 155 ptp4l[17645.466]: master offset -3388 s2 freq -21269 path delay 191 ptp4l[17646.466]: master offset -2317 s2 freq -21215 path delay 156 ptp4l[17647.466]: master offset -1362 s2 freq -20955 path delay 191 ptp4l[17648.466]: master offset -640 s2 freq -20641 path delay 191 ptp4l[17649.466]: master offset -203 s2 freq -20396 path delay 191 ... ptp4l[17660.466]: master offset -3 s2 freq -20190 path delay 234 ptp4l[17661.466]: master offset -3 s2 freq -20191 path delay 234 ptp4l[17662.466]: master offset 2 s2 freq -20187 path delay 235 ...
That actually just worked. The ConnectX-3 NICs were synchronized to within a few nanoseconds of each other after less than half a minute.
Now, I need to feed the correct time into the server’s PTP hardware clock with
-w to wait for
ptp4l to be ready,
-m to print to
-c is the network interface):
$ sudo phc2sys -s CLOCK_REALTIME -c cx3 -wm phc2sys[95992.748]: /dev/ptp0 sys offset -20572648 s0 freq +218028 delay 4440 phc2sys[95993.748]: /dev/ptp0 sys offset -20572410 s1 freq +218266 delay 4440 phc2sys[95994.748]: /dev/ptp0 sys offset -4 s2 freq +218262 delay 4450 phc2sys[95995.749]: /dev/ptp0 sys offset 1 s2 freq +218266 delay 4440 phc2sys[95996.749]: /dev/ptp0 sys offset 5 s2 freq +218270 delay 4450
Note that the
-d option is supposed to allow you to pass a PPS device, but
apparently, that’s only for disciplining
CLOCK_REALTIME with a PPS device from
the NIC. Therefore, I can’t use the PPS signal from my GPS directly. Instead, I
have to rely on NTP’s adjustments to
Now on the client, I’ll need to disable NTP to avoid it interfering with the PTP
signal. By default, Debian uses
systemd-timesyncd, which can be disabled via
sudo timedatectl set-ntp false. Otherwise, try either
sudo systemctl disable
--now ntpd.service or
sudo systemctl disable --now chronyd.service.
Now, I should be able to sync the client’s clock to it (
-c CLOCK_REALTIME is
-s is the network interface):
$ sudo phc2sys -s /dev/ptp1 -wm phc2sys[18917.156]: CLOCK_REALTIME phc offset -19059065 s0 freq -11778 delay 4381 phc2sys[18918.157]: CLOCK_REALTIME phc offset -19080139 s1 freq -32848 delay 4450 phc2sys[18919.157]: CLOCK_REALTIME phc offset -8422 s2 freq -41270 delay 4440 ... phc2sys[18929.158]: CLOCK_REALTIME phc offset 42 s2 freq -41245 delay 4420 phc2sys[18930.158]: CLOCK_REALTIME phc offset -2 s2 freq -41277 delay 4431 phc2sys[18931.158]: CLOCK_REALTIME phc offset 43 s2 freq -41232 delay 4450 phc2sys[18932.159]: CLOCK_REALTIME phc offset -35 s2 freq -41297 delay 4410 ...
Alright, it seemed to work! As you can see, it corrected a “massive” 19 ms error
on my PC’s clock (thanks,
systemd-timesyncd2) and moved it within some
nanoseconds of my server, whose clock is at least decently accurate due to the
PPS signal from the GPS. I checked the time on my system for sanity and it
appeared accurate, so I didn’t end up trashing my system clock.
There is a bit of jitter, so I am only comfortable with saying that this achieved sub-microsecond accuracy to the server, which is still pretty good. The server’s real-time clock should be accurate to within a few microseconds of UTC due to the PPS signal. Now I just need to daemonize this.
(Note: If you decide to stop here, remember to re-enable NTP.)
Now that I have
phc2sys working, I should make them services that
start automatically and use proper configuration files too.
For the server, I decided to use the following
ptp4l configuration file (at
/etc/linuxptp/ptp4l.conf, remember to replace
cx3 with your interface name):
[global] # Only syslog every 1024 seconds summary_interval 10 # Increase priority to allow this server to be chosen as the PTP grandmaster. priority1 10 priority2 10 [cx3] network_transport L2
For the client, I decided to the following configuration file:
[global] # Only syslog every 1024 seconds summary_interval 10 [cx3] network_transport L2
And the following
systemd unit (at
/etc/systemd/system/ptp4l.service) on both:
[Unit] Description=Precision Time Protocol service Documentation=man:ptp4l [Service] Type=simple ExecStart=/usr/sbin/ptp4l -f /etc/linuxptp/ptp4l.conf [Install] WantedBy=multi-user.target
Now just run
sudo systemctl enable --now ptp4l.service to start
phc2sys, the configuration file is pointless since we can’t put the
interface names into it. I decided to instead configure it with the
unit. On the server, I created
cx3 with your interface name,
-u 1024 means printing a summary every 1024
seconds to avoid log spam):
[Unit] Description=Synchronizing PTP clock to system time Documentation=man:phc2sys After=ptp4l.service [Service] Type=simple ExecStart=/usr/sbin/phc2sys -s CLOCK_REALTIME -c cx3 -w -u 1024 [Install] WantedBy=multi-user.target
On the client, I used the following
[Unit] Description=Synchronizing system time to PTP Documentation=man:phc2sys After=ptp4l.service [Service] Type=simple ExecStart=/usr/sbin/phc2sys -s cx3 -w -u 1024 [Install] WantedBy=multi-user.target
Now just run
sudo systemctl enable --now phc2sys.service to start
both machines. Now my PC should be synchronized as much as possible to the
server, which is synchronized to GPS time.
For fun, I decided to run PTP client on the Raspberry Pi 3 to see how well it syncs itself to my server. (This would also be rather indicative of what would happen if I tried to get a PPS time signal out of the Raspberry Pi 3.)
ptp4l doesn’t interact well with bridge interfaces or 802.3ad
bonds, so for this testing, I removed an interface from the 802.3ad link
aggregation I have to my switch.
To avoid interference with my existing setup, I am using PTP domain 1 for this
test (you don’t need
--domain=1 if it’s the only
ptp4l instance on your
LAN). I ran the following command on the server (where
rtl8111 is the
$ sudo ptp4l -2Smi rtl8111 --domain=1 ptp4l[170800.930]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[170800.930]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[170808.503]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES ptp4l[170808.503]: selected local clock [server EUI-64] as best master ptp4l[170808.503]: port 1: assuming the grand master role
Now on the Raspberry Pi, I ran:
$ sudo ptp4l -2Ssmi eth0 --domain=1 ptp4l[619306.404]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[619306.406]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[619307.984]: port 1: new foreign master [server EUI-64]-1 ptp4l[619311.984]: selected best master clock [server EUI-64] ptp4l[619311.985]: foreign master not using PTP timescale ptp4l[619311.985]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[619313.997]: master offset 832985 s0 freq -2743 path delay 177882 ... ptp4l[619329.998]: master offset 863399 s1 freq -842 path delay 177707 ptp4l[619330.998]: master offset -8896 s2 freq -1741 path delay 177707 ptp4l[619330.998]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[619331.998]: master offset 58508 s2 freq +5058 path delay 177707 ptp4l[619332.998]: master offset -7400 s2 freq -1540 path delay 178331 ... ptp4l[619429.004]: master offset 6511 s2 freq -35 path delay 174225 ptp4l[619430.004]: master offset 4366 s2 freq -245 path delay 174225 ptp4l[619431.004]: master offset 50330 s2 freq +4402 path delay 174225 ptp4l[619432.004]: master offset -6701 s2 freq -1308 path delay 181328 ptp4l[619433.004]: master offset -1725 s2 freq -812 path delay 182561 ptp4l[619434.004]: master offset -9388 s2 freq -1588 path delay 182561 ptp4l[619435.004]: master offset -14759 s2 freq -2140 path delay 182873 ptp4l[619436.004]: master offset -12987 s2 freq -1975 path delay 183020 ptp4l[619437.004]: master offset 8435 s2 freq +175 path delay 183020 ptp4l[619438.004]: master offset -7114 s2 freq -1387 path delay 183020
Note that in software timestamping mode,
ptp4l directly messes with the system
clock, and no
phc2sys is needed.
As we can see,
ptp4l struggled to make the offset fall under 10 μs due to the
software timestamping and the USB jitter. If I had used the Raspberry Pi 3 to
receive the PPS signal, this would be the best accuracy I could possibly
Now, I also have an Atomic Pi with a PCIe NIC (RTL8111), though it also lacks hardware timestamping. Let’s see how it fares:
$ sudo ptp4l -2Ssmi enp1s0 --domain=1 ptp4l[574.866]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[574.866]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE ptp4l[574.993]: port 1: new foreign master [server EUI-64]-1 ptp4l[578.993]: selected best master clock [server EUI-64] ptp4l[578.994]: foreign master not using PTP timescale ptp4l[578.994]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[580.031]: master offset 7185769 s0 freq +28159 path delay 38385 ... ptp4l[596.032]: master offset 7608532 s1 freq +54579 path delay 40711 ptp4l[597.032]: master offset -7046 s2 freq +53868 path delay 40711 ptp4l[597.032]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[598.032]: master offset -1398 s2 freq +54431 path delay 40711 ptp4l[599.032]: master offset -2483 s2 freq +54320 path delay 40985 ptp4l[600.032]: master offset 1492 s2 freq +54719 path delay 40985 ptp4l[601.032]: master offset 4112 s2 freq +54985 path delay 42100 ptp4l[602.032]: master offset 3030 s2 freq +54880 path delay 42100 ptp4l[603.032]: master offset -8799 s2 freq +53688 path delay 42983 ptp4l[604.032]: master offset 400 s2 freq +54609 path delay 42983 ptp4l[605.032]: master offset -119 s2 freq +54557 path delay 43305 ptp4l[606.032]: master offset -3942 s2 freq +54170 path delay 42853 ptp4l[607.032]: master offset 2211 s2 freq +54788 path delay 42403 ptp4l[608.032]: master offset 2455 s2 freq +54815 path delay 42853 ptp4l[609.032]: master offset -606 s2 freq +54508 path delay 43222 ptp4l[610.032]: master offset 490 s2 freq +54618 path delay 43222 ptp4l[611.032]: master offset -6645 s2 freq +53898 path delay 42237 ptp4l[612.032]: master offset 4334 s2 freq +55000 path delay 42089 ptp4l[613.032]: master offset 1950 s2 freq +54764 path delay 42089 ...
As you can see, it’s struggling less hard than the Raspberry Pi, but can only keep the clock within a few microseconds. This is fairly decent if you don’t have PTP-capable hardware.
Given that PTP struggles to work with 802.3ad link aggregation, it means that
I’d have to have a NIC dedicated to serving PTP on the server. This doesn’t seem
reasonable, so I won’t be deploying PTP for hosts other than my PC. As such,
converting the commands shown here into a configuration file for
left as an exercise for the reader.
With hardware timestamping NICs at both ends, it’s easy to achieve sub-microsecond accuracy with PTP.
With software timestamping NICs at both ends, it’s possible to achieve sub-10 μs accuracy with PTP.
With a Raspberry Pi, it’s possible to achieve sub-100 μs accuracy with PTP.
I hope this proved a useful and enjoyable read.
README.Debianfile has the following to say about
The service timemaster also isn’t enabled and started by default [sic]
That’s it. Not even a period to end the last sentence in the file. And the service is enabled by default even though it won’t start due to lack of
chronydin the default Debian install. It also mentions
phc2systhat don’t exist. ↩
To be fair,
systemd-timesyncdis designed to be lightweight and never designed to be more accurate than tens of milliseconds. It’s not like this level of precision is needed for normal desktop use. ↩