Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TimeHAT V2 Pi 5 HAT from OCP-TAP #674

Open
geerlingguy opened this issue Sep 20, 2024 · 28 comments
Open

Add TimeHAT V2 Pi 5 HAT from OCP-TAP #674

geerlingguy opened this issue Sep 20, 2024 · 28 comments

Comments

@geerlingguy
Copy link
Owner

geerlingguy commented Sep 20, 2024

OCP-TAP has announced the TimeHAT V2, a Pi 5 HAT that embeds an Intel i226 NIC along with a B-key M.2 slot for cards like GPS/GNSS adapters.

DSC01242

The M.2 slot routes the PPS signal straight to the i226 input, for GPS time synchronization.

The two SMA connectors on the HAT can be configured for PPS in/out (IIRC the Pi 5 only has one PPS pin on it's built-in 1 Gbps NIC, but it is not exposed externally).

The preliminary schematics are available on the OCP-TAP GitHub here, and I have asked if they're considering selling units (or working with someone else to do so), and it sounds like this is a possibility!

From the LinkedIn post at the top of this issue, they also showed a demo of < 10ns sync using PTP with ptp4l:

1723338953158

Here is the TimeHat (open source and available on https://lnkd.in/gWNDJtDn) an add-on for the Raspberry Pi 5 that packs the Intel i226 with two SMA connectors, configurable to do PPS in/out (a feature that is unfortunately missing on the RPi 5). In addition the TimeHat got a connector for the latest RCB m.2 form factor to host a GNSS receiver by routing its PPS signal to the i226 as an input allowing it to become a Time Server for both NTP and PTP.

In this setup two Raspberry Pi 5 boards, each with a TimeHat are getting synchronized (the one on top acts as a time server and the one on the bottom as client) via PTP and both provide a PPS output that is compared (one act as the trigger and one gets measured) on the oscilloscope. The first sigma of the time sync error is under 10ns.

See related: https://pipci.jeffgeerling.com/hats/ocp-tap-timehat.html / #619

Setup Instructions

Note: Still working on getting this running 100%, but here's how to get all the interfaces going for now:

First, add the following lines at the end of /boot/firmware/config.txt:

# For i226 compatibility, see https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/674
dtoverlay=pciex1-compat-pi5,mmio-hi

# To allow the u-blox GPS to work
dtparam=uart0=on

Then, make sure you remove the part console=serial0,115200 from /boot/firmware/cmdline.txt so the u-blox GPS can use /dev/ttyAMA0 for it's serial output.

Also, if you are on an older Pi OS or firmware, run sudo apt update && sudo apt full-upgrade, or consider running sudo rpi-update to get everything to the latest cutting edge releases.

To get the I226-LM NIC working, make sure it's plugged into the Pi's PCIe FFC connection properly, and check if it's recognized with lspci:

pi@time-pi:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-LM (rev 03)
...

Check if it gets a connection with ethtool eth1:

pi@time-pi:~ $ ethtool eth1
Settings for eth1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	                        2500baseT/Full

If you're having trouble with DHCP, you may need to configure a manual IP address, using nmtui. I've also found that 2.5 Gbps connections don't work at all, and you can't force 1 Gbps speeds on a 2.5 Gbps connection with the igb driver that ships in Pi OS's kernel, so right now I just make sure I only plug in the HAT's Ethernet port to 1 Gbps ports on my switch.

Install GPS software to work with the GPS modem:

TODO

Configure gpsd and NTPsec to sync via PPS. (TODO)

@geerlingguy
Copy link
Owner Author

I was sent two of these (along with an M.2 u-blox ZED-F9T-00B-01 module, capable of high-accuracy GPS reception.

I hope to get this set up as a time server in my rack soon—maybe I can also work with Masterclock to set up a nice time display as well?

@geerlingguy
Copy link
Owner Author

lol, how apropos ShortCircuit just posted a video on the Open Timecard Mini today.

@geerlingguy
Copy link
Owner Author

Just booted a new Pi OS install, but for some reason the card is not showing up when I run lspci. I'm wondering if the cable orientation is off if you have it sitting as a HAT (I noticed the photos on the LinkedIn have a cable that's basically flipped opposite of what it would be if you have the HAT on top of the Pi... I think).

@geerlingguy
Copy link
Owner Author

Following along with #606 (where we were debugging issues with the i225-V adapter), I've done the following:

  1. Run sudo rpi-update
  2. Add dtoverlay=pciex1-compat-pi5,mmio-hi to my /boot/firmware/config.txt

After a reboot, I still see:

pi@pi5:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge

(This is also with dtparam=pciex1 in my /boot/firmware/config.txt file.)

@geerlingguy
Copy link
Owner Author

Aha! I was using the little Pineboards FFC with the arrow on the GPIO side of the connection (mostly because I saw there was a tiny triangle on that side of the connector on the HAT, and I love matching triangles:

DSC01246

(In the photo above I just placed the cable atop the other one in the orientation I had it.)

I spun the cable around (did a 180) so the triangle is on the other side:

DSC01245

After booting up, now I see the card:

pi@pi5:~ $ sudo lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-LM (rev 03)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 21)
0001:01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge

@geerlingguy
Copy link
Owner Author

...surprisingly, it looks like the Intel IGC driver is already present?

pi@pi5:~ $ dmesg | grep igc
[    5.532414] igc 0000:01:00.0: enabling device (0000 -> 0002)
[    5.532448] igc 0000:01:00.0: PCIe PTM not supported by PCIe bus/controller
[    5.592460] igc 0000:01:00.0 (unnamed net_device) (uninitialized): PHC added
[    5.643468] igc 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    5.643481] igc 0000:01:00.0 eth1: MAC: 00:a0:c9:00:00:01
pi@pi5:~ $ sudo lspci -vvv
...
0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-LM (rev 03)
	Subsystem: Intel Corporation Ethernet Controller I226-LM
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 128 bytes
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at 1b80000000 (32-bit, non-prefetchable) [size=1M]
	Region 3: Memory at 1b80100000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 512 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [140 v1] Device Serial Number 00-a0-c9-ff-ff-00-00-01
	Capabilities: [1c0 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [1f0 v1] Precision Time Measurement
		PTMCap: Requester:+ Responder:- Root:-
		PTMClockGranularity: 4ns
		PTMControl: Enabled:- RootSelected:-
		PTMEffectiveGranularity: Unknown
	Capabilities: [1e0 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=55us PortTPowerOnTime=70us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=55us LTR1.2_Threshold=131072ns
		L1SubCtl2: T_PwrOn=70us
	Kernel driver in use: igc
	Kernel modules: igc

And it's giving me 2.5 Gbps full duplex:

pi@pi5:~ $ sudo ethtool eth1
Settings for eth1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	                        2500baseT/Full
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	                        2500baseT/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 2500Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	MDI-X: off (auto)
	Supports Wake-on: pumbg
	Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes

@geerlingguy
Copy link
Owner Author

geerlingguy commented Sep 20, 2024

In dmesg, I see the following when I connect a cable:

[  444.905640] igc 0000:01:00.0 eth1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX

But I don't get an IP address assigned:

pi@pi5:~ $ ip a
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:a0:c9:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ec7b:15ad:149:c0b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

The interface appears in nmtui, and all the settings look correct (auto negotiate an IPv4/IPv6 address, etc.), and if I go to 'Activate' it though, it times out after a while with:

Could not activate connection:
Activation failed: IP configuration
could not be reserved (no available
address, timeout, etc.)

@geerlingguy
Copy link
Owner Author

Exploring PTP a bit:

pi@pi5:~ $ sudo apt install linuxptp

pi@pi5:~ $ ethtool -T eth1
Time stamping parameters for eth1:
Capabilities:
	hardware-transmit
	software-transmit
	hardware-receive
	software-receive
	software-system-clock
	hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
	off
	on
Hardware Receive Filter Modes:
	none
	all

pi@pi5:~ $ sudo ptp4l -i eth1 -m
ptp4l[938.535]: selected /dev/ptp0 as PTP clock
ptp4l[938.536]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[938.536]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[946.257]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[946.257]: selected local clock 00a0c9.fffe.000001 as best master
ptp4l[946.257]: port 1: assuming the grand master role
...

Red Hat has some decent docs on Configuring PTP using ptp4l.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Sep 20, 2024

To configure PPS signals from the NIC, I should be able to use the testptp software. Going to have to wrap up my testing today but will get back at it soon, and try to sync across two Pi 5s with these TimeHATs!

sudo testptp -d /dev/ptp0 -L2,0
sudo testptp -d /dev/ptp0 -e -1

@geerlingguy
Copy link
Owner Author

I tested the other HAT that was shipped (both have a little EEPROM for the NIC that maybe dictates how DHCP works. It also doesn't retrieve an IPv4 address over DHCP, so I instead set a static IP.

  1. sudo nmtui
  2. Select 'Edit a connection'
  3. Select 'Wired connection 2'
  4. Set 'IPv4 CONFIGURATION' to 'Manual'
  5. Add an IP address, Gateway, and DNS server.
  6. Exit to save the settings

Now the NIC is getting the IP I set:

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:a0:c9:00:00:02 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.98/8 brd 10.255.255.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::2e62:805:2ffc:c29c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

But I can't actively use that network connection—it just times out if I unplug the main Pi NIC. I may try recompiling the kernel with the latest i226-V driver...

@geerlingguy
Copy link
Owner Author

geerlingguy commented Sep 26, 2024

Regarding the u-blox: It should be accessible via /dev/ttyAMA0 at 115200 baud:

tio -b 115200 /dev/ttyAMA0

Ctrl-t, then q to exit.

(Except I'm not seeing ttyAMA0 by default.)

Checking with i2c-tools after enabling the I2C interface with sudo raspi-config:

pi@pi5:~ $ sudo apt install -y i2c-tools
pi@pi5:~ $ i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- 42 -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --     

I used sudo raspi-config to disable the Serial Port login shell, and enable the serial interface (dtparam=uart0=on in /boot/firmware/config.txt).

After rebooting, I'm getting GPS module data over serial!

pi@pi5:~ $ tio -b 115200 /dev/ttyAMA0
[17:04:08.831] tio v2.5
[17:04:08.831] Press ctrl-t q to quit
[17:04:08.832] Connected
$GNRMC,,V,,,,,,,,,,N,V*37
$GNVTG,,,,,,,,,N*2E
$GNGGA,,,,,,0,00,99.99,,,,,,*56
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,1*33
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,2*30
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,3*31
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,4*36
$GPGSV,1,1,00,*55
$GPGSV,1,1,00,*55
$GLGSV,1,1,00,*49
$GLGSV,1,1,00,*49
$GAGSV,1,1,00,*44
$GAGSV,1,1,00,*44
$GBGSV,1,1,00,*47
$GBGSV,1,1,00,*47
$GNGLL,,,,,,V,N*7A
$GNRMC,,V,,,,,,,,,,N,V*37
$GNVTG,,,,,,,,,N*2E
$GNGGA,,,,,,0,00,99.99,,,,,,*56

I'm getting NMEA messages, but right now I'm connected to an antenna inside my studio, which is probably not going to get a signal. Need to find a window somewhere, heh.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Sep 26, 2024

Looking around at GPS antenna and cabling guidelines, the first DDG result was from Masterclock! GPS Antenna & Cable Information. They recommend four standardized antenna options.

@geerlingguy
Copy link
Owner Author

Just noting PTP works out of the box on CM5 (no HAT required ;)

pi@cm5:~ $ ls /dev/ptp*
/dev/ptp0  /dev/ptp1

pi@cm5:~ $ cat /sys/class/ptp/ptp0/clock_name
bcm_phy_ptp
pi@cm5:~ $ cat /sys/class/ptp/ptp1/clock_name
gem-ptp-timer

pi@cm5:~ $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
    hardware-transmit
    hardware-receive
    hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
    off
    on
    onestep-sync
    onestep-p2p
Hardware Receive Filter Modes:
    none
    ptpv2-event

And testing with testptp:

pi@cm5:~/Downloads $ wget https://raw.githubusercontent.com/raspberrypi/linux/refs/heads/rpi-6.6.y/tools/testing/selftests/ptp/testptp.c

pi@cm5:~/Downloads $ sudo ./testptp -d /dev/ptp0 -L0,2
set pin function okay
pi@cm5:~/Downloads $ sudo ./testptp -d /dev/ptp0 -p 1000000000
periodic output request okay

@geerlingguy
Copy link
Owner Author

mDNS doesn't seem to be working on the i226 interface by default either, but possibly just because that interface isn't working in general, heh...

The configuration currently:

pi@time-pi:/etc/NetworkManager/system-connections $ sudo !!
sudo cat 'Wired connection 2.nmconnection' 
[connection]
id=Wired connection 2
uuid=c26ef42c-3136-3a6b-af4e-0c64891e346d
type=ethernet
autoconnect-priority=-999
interface-name=eth1

[ethernet]

[ipv4]
address1=10.0.2.98/8,10.0.2.1
dns=10.0.2.1;
method=manual

[ipv6]
addr-gen-mode=default
method=auto

[proxy]

@geerlingguy
Copy link
Owner Author

After seeing Connection loss with Intel Ethernet Controller I226-V, I added pcie_aspm=off to /boot/firmware/cmdline.txt, and that doesn't seem to help.

I tried disabling WiFi with nmcli radio wifi off and then just lost the connection. ping doesn't give me any response either.

@geerlingguy
Copy link
Owner Author

Well that's fun... after reading through various forum topics about the i226-V, I found many people had connection issues (often flaky, some resolved by disabling ASPM, others not...), and since that didn't help for me, I tried the next option:

Plugging the Pi into a 1 Gbps port instead of 2.5 Gbps.

And now it's working perfectly—even both LEDs on the connector (green for activity and amber for connection) are lit up now.

So... some weird bug in igc with 2.5 Gbps speeds?

You can force a speed for an interface with:

# Open nmcli's connection editor
sudo nmcli c edit "Wired connection 2"

# Inside nmcli's connection editor...
goto ethernet
set auto-negotiate no
set speed 1000
set duplex full
back
save persistent
quit

(Thanks to this guide for the connection speed instructions.)

This results in the following configuration:

pi@time-pi:~ $ sudo cat /etc/NetworkManager/system-connections/Wired\ connection\ 2.nmconnection 
[connection]
id=Wired connection 2
uuid=c26ef42c-3136-3a6b-af4e-0c64891e346d
type=ethernet
autoconnect-priority=-999
interface-name=eth1
timestamp=1733870503

[ethernet]
duplex=full
speed=1000

[ipv4]
address1=10.0.2.98/8,10.0.2.1
dns=10.0.2.1;
method=manual

[ipv6]
addr-gen-mode=default
method=auto

[proxy]

@geerlingguy
Copy link
Owner Author

Oddly, when I plug it into a 2.5 Gbps port again, I get:

pi@time-pi:~ $ nmcli con show 'Wired connection 2'
...
802-3-ethernet.speed:                   1000
802-3-ethernet.duplex:                  full

However, ethtool reports:

	Speed: 2500Mb/s
	Duplex: Full

And I'm not getting packets through it (with the same single orange light on the port).

Of course... now that I look closely in dmesg:

[  476.900511] igc 0000:01:00.0 eth1: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
[  476.902744] igc 0000:01:00.0 eth1: Force mode currently not supported

And searching for that turns up this gem of a thread: i225 - IGC linux kernel Driver - Force Speed and duplex not supported, in which Intel support reps keep trying to pass the buck on an obvious driver shortcoming, resulting in:

If I understand correctly, everyone is passing the buck and I'm like an idiot with this bleep card!

That's about how I feel too haha. It seems like the igc driver is... less than ideal.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 10, 2024

Visualizing some GPS data...

# Install prerequisites
sudo apt-get install gpsd gpsd-clients

# Configure gpsd - edit `/etc/default/gpsd` and set these options:
DEVICES="/dev/ttyAMA0"
GPSD_OPTIONS="-s 115200 -n"

# Restart gpsd and make sure it's enabled at boot
sudo systemctl restart gpsd
sudo systemctl enable gpsd

# ALTERNATIVELY: Stop gpsd and launch it manually
sudo gpsd /dev/ttyAMA0 -s 115200 -n -F /var/run/gpsd.sock

# Display GPS info (choose one)
gpsmon -n
cgps -s

After a reboot, I'm not getting GPS modem information now, lol... have to debug that tomorrow!

Ah, I had gpsd running when I quickly checked with tio -b 115200 /dev/ttyAMA0, and gpsd was taking over that output. All is well. I've placed my antenna into a window and will let it go for a few hours and check if I get a position lock at some point. Right now still spitting out:

$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,1*33
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,2*30
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,3*31
$GNGSA,A,1,,,,,,,,,,,,,99.99,99.99,99.99,4*36

And cgps is giving:

┌───────────────────────────────────────────┐┌──────────────────Seen  0/Used  0┐
│ Time:          n/a (0)                    ││GNSS   PRN  Elev   Azim   SNR Use│
│ Latitude:        n/a                      ││                                 │
│ Longitude:       n/a                      ││                                 │
│ Alt (HAE, MSL):        n/a,       n/a     ││                                 │
│ Speed:           n/a                      ││                                 │
│ Track (true, var):                n/a deg ││                                 │
│ Climb:           n/a                      ││                                 │
│ Status:         NO FIX (0 secs)           ││                                 │
│ Long Err  (XDOP, EPX):  n/a ,  n/a        ││                                 │
│ Lat Err   (YDOP, EPY):  n/a ,  n/a        ││                                 │
│ Alt Err   (VDOP, EPV):  n/a ,  n/a        ││                                 │
│ 2D Err    (HDOP, CEP):  n/a ,  n/a        ││                                 │
│ 3D Err    (PDOP, SEP):  n/a ,  n/a        ││                                 │
│ Time Err  (TDOP):       n/a               ││                                 │
│ Geo Err   (GDOP):       n/a               ││                                 │
│ ECEF X, VX:              n/a    n/a       ││                                 │
│ ECEF Y, VY:              n/a    n/a       ││                                 │
│ ECEF Z, VZ:              n/a    n/a       ││                                 │
│ Speed Err (EPS):        n/a               ││                                 │
│ Track Err (EPD):        n/a               ││                                 │
│ Time offset:            n/a               ││                                 │
│ Grid Square:            n/a               ││                                 │
└───────────────────────────────────────────┘└─────────────────────────────────┘

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 11, 2024

Well, now I'm getting data when observing with tio -b 115200 /dev/ttyAMA0, but gpsmon /dev/ttyAMA0 and gpsd/cgps are blank. Though it looks like gpsd and gpsmon are negotiating 9600 8N1:

time-pi:/dev/ttyAMA0 9600 8N1 Unknown device> No device defined yet

Apparently it was the datarate. autobaud was selecting 9600, and if I specified 115200 manually, everything magically started working:

sudo gpsd /dev/ttyAMA0 -s 115200 -N -D3 -F /var/run/gpsd.sock

So now to get things launched in the background manually, I run:

sudo gpsd /dev/ttyAMA0 -s 115200 -F /var/run/gpsd.sock
cgps

And I get:

Screenshot 2024-12-11 at 9 54 25 AM

(Note: I've updated the above install instructions so I can use the daemonized gpsd which is easier to manage.)

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 11, 2024

I've been working on getting PPS data out, as I do see a time signal, and it's updating every second with offsets and such... but right now I see nothing with ppscheck:

pi@time-pi:~ $ sudo ppscheck /dev/ttyAMA0
# Seconds  nanoSecs   Signals
^C

gpsmon is showing all the satellite data, but is saying PPS: N/A:

Screenshot 2024-12-11 at 10 35 37 AM

@geerlingguy
Copy link
Owner Author

@ahmadexp recommended using the PHC (Physical Hardware Clock) of the i226 for the source, and then use Chronyd for NTP server. That way the NIC's PHC would be the source for both PTP and NTP.

And though some references to i225/i226 indicate it can't do PPS out, this Intel community post shows it working on i225-T1, so I think it's a matter of the documentation lacking, rather than hardware. Also, tons of cards don't expose the pins at all, so that's also an issue :)

But on there:

# Configure SDP0 pin for periodic output
sudo ./testptp -d /dev/ptp1 -L 0,2
# Generate pulse of width 1us every second
sudo ./testptp -d /dev/ptp1 -p 1000000000

On my Pi setup, the i226 is eth1, and I can grab the PTP clock info:

$ sudo ethtool -T eth1 | grep PTP
PTP Hardware Clock: 0

Then I can configure /dev/ptp0:

# Enable PPS out
testptp -d /dev/ptpX -L0,1  # Might be -L2,1 on TimeHat V2?

# Enable PPS in
testptp -d /dev/ptpX -e -1

I'm not sure if I can configure both on separate pins, but I may test that later. This is all using OCP testptp.c fork, which is forked off the Linux kernel testptp.c.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 12, 2024

Setting up Chrony, first just checking on current system time:

pi@time-pi:~ $ timedatectl
               Local time: Thu 2024-12-12 09:17:10 CST
           Universal time: Thu 2024-12-12 15:17:10 UTC
                 RTC time: Thu 2024-12-12 15:17:10
                Time zone: America/Chicago (CST, -0600)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Install and configure Chrony:

# Install and enable chrony at boot.
sudo apt install -y chrony
sudo systemctl start chrony.service
sudo systemctl enable chrony.service

The chrony config is in /etc/chrony/chrony.conf.

Need to allow clients to request sync from the server, so add the local IP range to a separate config file /etc/chrony/conf.d/allow.conf, then restart chrony:

# Allow time sync requests from clients on local network (port 123/UDP).
allow 10.0.2.0/24

Exploring the default configuration:

pi@time-pi:~ $ chronyc sources -v  # use `-n` for terse output

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^- ip212-227-240-160.pbiaas>     2   6    17    42  -4785us[-4886us] +/-   64ms
^- blotch.image1tech.net         2   6    17    41  +1462us[+1361us] +/-   83ms
^* srv2.maxhost.io               2   6    27    40  +2223us[+3967us] +/-   26ms
^+ h135-134-111-122.mdsnwi.>     2   6    17    43  -3999us[-2255us] +/-   23ms

pi@time-pi:~ $ chronyc -n tracking
Reference ID    : D842302A (216.66.48.42)
Stratum         : 2
Ref time (UTC)  : Thu Dec 12 15:31:10 2024
System time     : 0.000519845 seconds fast of NTP time
Last offset     : +0.001082936 seconds
RMS offset      : 0.001838107 seconds
Frequency       : 0.107 ppm slow
Residual freq   : +1.662 ppm
Skew            : 11.269 ppm
Root delay      : 0.075731449 seconds
Root dispersion : 0.001926737 seconds
Update interval : 65.0 seconds
Leap status     : Normal

I am trying to figure out how to get Chrony to use the PHC from the NIC now... see Does chrony support PTP? and How can I avoid using wrong PHC refclock?.

Edit: Ah, found Server using reference clock on NIC. Jotting down the configuration I'm testing here:

# TODO: Trying to get the right values for the i226 on the TimeHat V2...
local stratum 1
refclock PHC /dev/ptp0:extpps:pin=0 dpoll -4 poll -2 rate 16 width 0.03125 refid GPS lock NMEA maxlockage 32
refclock SHM 0 refid NMEA noselect offset 0.120 poll 6 delay 0.010
hwtimestamp * minpoll -4

I think I should add my config into /etc/chrony/conf.d/refclock.conf

I at least know the time serving portion of Chrony is working well. From another Pi on the network, via WiFi (since time-pi.local is currently running in my car to get a GPS lock, lol):

# Installed `ntpdate` with `sudo apt install -y ntpdate`
$ ntpdate -q 1.debian.pool.ntp.org
2024-12-12 09:43:38.11412 (-0600) +0.022135 +/- 0.040153 1.debian.pool.ntp.org 172.234.37.140 s2 no-leap
$ ntpdate -q 10.0.2.236
2024-12-12 09:43:48.557480 (-0600) +0.063267 +/- 0.066152 10.0.2.236 s2 no-leap

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 12, 2024

Going to test with the following inside /etc/chrony/conf.d/refclock.conf:

local stratum 1
refclock PHC /dev/ptp0 tai poll 0 trust
hwtimestamp *

After a restart (sudo systemctl restart chrony), I see in its output:

Dec 12 13:25:52 time-pi chronyd[2040]: Enabled HW timestamping on eth1
Dec 12 13:25:52 time-pi chronyd[2040]: Frequency 0.649 +/- 0.150 ppm read from /var/lib/chrony/chrony.drift
Dec 12 13:25:52 time-pi chronyd[2040]: Using right/UTC timezone to obtain leap second data
Dec 12 13:25:52 time-pi chronyd[2040]: Loaded seccomp filter (level 1)
Dec 12 13:25:52 time-pi systemd[1]: Started chrony.service - chrony, an NTP client/server.
Dec 12 13:25:54 time-pi chronyd[2040]: Selected source PHC0
Dec 12 13:25:54 time-pi chronyd[2040]: System clock wrong by -62418.617578 seconds
Dec 11 20:05:36 time-pi chronyd[2040]: System clock was stepped by -62418.617578 seconds
Dec 11 20:05:36 time-pi chronyd[2040]: System clock TAI offset set to 37 seconds
Dec 11 20:05:40 time-pi chronyd[2040]: Source 23.150.41.122 replaced with 216.82.45.133 (2.debian.pool.ntp.org)

However, the date being returned is a bit off... comparing to Debian NTP server:

$ ntpdate -q 10.0.2.236
2024-12-11 20:06:41.822120 (-0600) -62418.569777 +/- 0.042449 10.0.2.236 s1 no-leap
$ ntpdate -q 1.debian.pool.ntp.org
2024-12-12 13:27:14.575574 (-0600) +0.004811 +/- 0.017758 1.debian.pool.ntp.org 67.217.246.204 s2 no-leap

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 12, 2024

Exploring syncing the system clock...

  • ptp4l config is in /etc/linuxptp/ptp4l.conf — this is used to designate interfaces being grandmaster (e.g. masterOnly) or not
  • phc2sys -a -rr will attempt to synchronize the system clock from a clock source automatically...
  • Check on current status with sudo pmc 'get PARENT_DATA_SET' 'get CURRENT_DATA_SET' 'get PORT_DATA_SET' 'get TIME_STATUS_NP' -u -b 0 (currently returns no output for each of the commands on my Pi)

There's a nice LinuxPTP guide on an audiophile forum that spells things out in some detail:

LinuxPTP package contains ptp4l, phc2sys, and pmc. The ptp4l is used for initialization, sanitize HW clocks on NICs and decides the role (MC, BC or TC). The phc2sys is used for synchronization between system clock and NIC's HW clock. The pmc is used for function check / verification.

But to get chrony going with GPS time for the time being (heh), I'm glancing through this guide for GPS + chrony on Pi:

pi@time-pi:~ $ cat /etc/chrony/conf.d/refclock.conf 
local stratum 1
refclock PHC /dev/ptp0 tai poll 0 trust
refclock SHM 0 offset 0.5 delay 0.2 refid NMEA
hwtimestamp *

pi@time-pi:~ $ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
#* PHC0                          0   0   377     1     -3ns[  -13ns] +/-   55ns
#x NMEA                          0   4   377    15  -62419s[-62419s] +/-  100ms
^x hc-007-ntp1.weber.edu         2   6   377    29  -62419s[-62419s] +/-   65ms
^x ntp4.lonet.org                2   6   377    29  -62419s[-62419s] +/-   43ms
^x h96-60-160-227.stgrut.br>     1   6   377    28  -62419s[-62419s] +/-   34ms
^x 44.190.5.123                  2   6   377    29  -62419s[-62419s] +/-   34ms

Now I'm just wondering - how do I set the PHC0 time from NMEA, so it's the correct source of truth?

@geerlingguy

This comment was marked as outdated.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 12, 2024

Okay, after a nice discussion with @leoleovich, I realized I was kind of looking at things the wrong way.

My overall architecture should be:

  1. GPS gets time data from stratum 0 clock
  2. gpsd manages the GPS data and feeds it into shared memory (SHM)
  3. chrony is configured to poll GPS data in addition to Internet time servers, as configured in /etc/chrony config files. If using GPS, no need to configure a reference to a PHC (Physical Hardware Clock) on the NIC that's running PTP
    - chrony is also configured (with the allow setting) to accept incoming requests on UDP port 123 for NTP time sync, at least in my case!
  4. PTP on this Grandmaster Pi get it's time from the system clock using phc2sys.
  5. ptp4l then sets up PTP hardware timestamping to synchronize from this Pi out to other computers on the network via PTP.

So NTP clients will get their time served through chrony. chrony also takes care of keeping the system clock in sync with GPS time. Then phc2sys synchronizes the system clock to the PHC on the NIC for PTP timestamping.

If I wanted to configure chrony to use the PHC on the NIC on another Pi as the time source, I would add:

refclock PHC /dev/ptp0 tai poll 0 trust
hwtimestamp *

@geerlingguy
Copy link
Owner Author

After allowing things to run a while, it looks like the GPS signal is "x", for "may be in error". Need to check on why. It could be reflections since the antenna is currently inside my building (though getting a 3D position lock... which moves around lol).

pi@time-pi:~ $ chronyc sources -v

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
#x GPS                           0   4   377    22   +104ms[ +104ms] +/-  360us
^* 23.155.40.38                  1   9   371    81   +958us[+1517us] +/-   14ms
^+ 65-100-46-166.dia.static>     1   9   377   286  +2708us[+3239us] +/-   36ms
^- dns-e.ns4v.icu                2  10   377   686  +7192us[+7669us] +/-   65ms
^+ t2.time.bf1.yahoo.com         2  10   377   619  -1076us[ -590us] +/-   18ms

Also working on automating setup in an Ansible playbook now. I'd like to make this Pi able to be replaced in a moment, just by running the playbook.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 13, 2024

Now that I have the time server working—mostly—I'm putting all the configuration in an Ansible playbook: https://github.com/geerlingguy/time-pi

I'll be working more on the software side there, and if there are any other hardware notes, I'll drop them in this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant