I cannot find any source files for the FPGA code in the repository (no .v .vhd .vhdl files and I can't see anything by looking around manually). The FPGA directory [1] contains some binaries and some documents relating to commercial code from a Swiss firm [2].
I've seen plenty of projects described as open source when the firmware is open source and the hardware is closed but this is the first time I've seen one where the schematics and board layout are open but the firmware is closed. Note that the hardware itself is quite simple, the smart stuff happens in three modules: GNSS receiver, precision clock and FPGA. To me, the contents of the FPGA source code are the only interesting part of this project. Additionally, you won't be able to meaningfully modify or reuse this project without editing the source code.
The hardware module these FPGA binaries seem to be compiled for is described as an AC7100B, the source given for these is a defunct ebay link [3] [4] :-O This project uses two and half grand's worth of atomic clock and the heart of it runs on a module that fell off the back of a lorry!?
Edit: I realise this comes across as quite negative. This looks like a neat project, one I would actually use, but only if it were meaningfully open source.
It's a particular module for the Xilinx Artix 7 XC7A100T, made by a Shanghai-based integrator. As an individual user, you could buy these on Aliexpress, from eBay resellers, or from Amazon resellers. Facebook presumably has an arrangement with the contract manufacturer that's not going to be possible for an individual developer ordering ones and twos to replicate.
However, to modify the HDL, you'd need a $3,000 license for Xilinx Vivado, which is pretty much a nonstarter for amateurs.
Thank you for id'ing the module. I can now see how using this would work for Facebook's needs.
I think the XC7A100T is supported by the free (and limited) edition of Vivado [1]. Of course, the confusion about what you can and can't do with the HDL is in itself a barrier to open source and hobby work. This is true even when the work is theoretically possible and everyone has the best of intentions. <sigh>
That is not true. First of all, the reason why we used the Alinx module (which you can buy of ebay as well as Amazon), was to make it easier for the community to manufacture and build. Our internal version of the time card does not include the module!
The second piece about the XC7A100T, you do not need any license to do any development as the free version of vivado is more than enough to work with.
Give us a bit of time and we will have a fully functional open HDL ready for you to add things.
It is very easy to sit down and nag about a project. I want you to understand that I had only one goal in mind when I released this project to public and that was allowing others to have a head start to work on time related project. Of course, I work for Facebook and I try to help my company with new technologies as a researcher, but helping the community and releasing the source code (to whatever I had) is just a core value of how I love to support the community.
you don't need vivado to modify the hdl, you need vivado (or some other proprietary xilinx toolchain) to produce the bitstream which actually deploys to the fpga.
That's a level of nitpicking that I didn't bother with.
Of course the Verilog or VHDL is just plaintext, which you can modify anywhere, but to actually do anything with it you need to compile it into a bitstream.
It does make me wonder if someone could set up a CI server to get around proprietary compilers like Vivado. Point the community resource at your source and get back a binary! Obviously you'd need to strip the license ID...
where if you wanted a tracker that wasn't a full controller you had to build it yourself. This involved a paid workshop in Seattle to get access to the dev kit and learn their tooling. Hardware was open source, but firmware was closed. If you wanted firmware changes you had to pay Synapse consulting fees for them to do it for you, and the updated firmware would then be made available free of charge to anyone else who paid for the workshop.
The Valve situation sounds like they attempted to open source after the fact, which doesn’t work with large hardware projects.
The only way to really open source a hardware project of this magnitude is to set open source as a constraint from the start. If you don’t, you inevitably end up with someone licensing some code somewhere from a vendor that can’t be open sourced. It gets integrated deeply enough that the project can’t exist without that piece and it’s too late to rewrite around it so the open source commitment goes out the window.
If I had to guess, it sounds like Valve tried to work around the situation with the workshop where participants entered into some contract as part of the workshop.
And if I had to guess about the Facebook situation, it was probably decided to open source the project after the fact. I’m guessing they wrote the website copy while someone else went off to try to secure open source permission for all components involved and they never reconciled the two efforts, hence the weird dead eBay links and missing files.
I shouldn't have called it open source, it was source available to customers who paid for the training course and signed a license agreement. Don't think they ever had intention of making it open. I found the model especially strange because of the requirement that any custom changes you asked for would be available to everyone else (your competitors). I guess they didn't actually want the workload of custom changes so this was a way of further disincentivizing it?
> I found the model especially strange because of the requirement that any custom changes you asked for would be available to everyone else (your competitors).
It's the same model as any game-console SDK. If you ask for changes to the SDK, those changes are going into "the SDK", which all customers get access to. Platform providers don't tend to offer work-for-hire for custom forks of their code; instead, what they're offering is to prioritize adding certain features to the shared upstream codebase. In the end, the platform provider wants there to just be one SDK, that solves the superset of all their customers' problems. Anything else is a ridiculously-large maintenance burden.
Just give us a bit of time. We had to use some IPs that due to licensing limitations prohibited us to release the source and we were only allowed to release the bin files. We are replacing them to open source cores and we will have that part of the project available to you too. I hope you understand the situation of the project. I would love to see people adding more features to the FPGA part of it such as IRIG-B output and many other cool things that can be done. Please stay tuned.
However, I think they've used code from NetTimeLogic under a licence [1][2] that precludes distributing the source:
> 3.2 Distribution Rights. LICENSEE may reproduce and distribute the Licensed Materials, solely in binary form that operates in LICENSEE’s system-level hardware products.
This is a perfectly reasonable way of developing a system but it is sadly incompatible with sharing your work as open source.
I have no problem with this project either and find it interesting, but I agree with you completely that it looks like someone took a hobby-level experiment with a corporate budget to the prototype stage then realized it had near zero practical application so, "Open source!"
What's interesting to me is how the form factor of timing GPS modules has stayed constant over the years. I started my GNSS timing journey with a used Trimble GPS from the 2000s, and it has the same form factor and pinout as the modern multi-constellation timing GPSes from uBlox.
I've had a GPS clock going for several years at this point, and without an atomic clock or really any fanciness (just LinuxPPS and Chrony), I see about +/- 380ns, which is pretty good. NTP to the Internet gives me jitter in the range of about 20ms-70ms, about 5 orders of magnitude worse.
(The version a few iterations ago looked like this: https://github.com/jrockway/beaglebone-gps-clock. But I now have a uBlox multi-constellation GPS, which is much more accurate with my limited view of the sky from my Brooklyn apartment. And I 3D printed the case, so it actually looks presentable instead of like some crazed madman that attacked a plastic case with a hacksaw -- which is exactly how I made the first case. As for the DS3231 RTC that I added... that seems to be stable within about 1.5us, which is pretty impressive. I tuned it a little bit with the trim register, though.)
My takeaway from this article is that the project seems quite compelling, but a lot of cost is added to support PTP. I see why Facebook needs that, but to sync time precisely throughout my lab, all I need is a coaxial cable with 10MHz on it. I don't mind if my workstation is 20ms off UTC.
In the Zoom Age, ambiently glanceable accurate (for humans) time is super useful, as is the temporal pie chart of time left vs. how much I have to care how much time is left illustrated by analog clock faces.
Nice project, I've been working on something similar, but with an Arduino (Adafruit ESP32), an Adafruit GPS featherwing and a RTC. I originally started with a Linux-based solution like yours but no matter what I did I couldn't get the accuracy below about 5ms and my goal is sub-ms (verified with an oscilloscope.)
Yeah, I think you're on the right track. I like less and less having a Linux box to maintain, and there is less control over the timing of internal operations compared to a microcontroller, of course. On the other hand, Chrony is really, really good, and it would be a lot of code and debugging to write something that good from scratch.
With the Beaglebone, it's theoretically possible to drive the oscillator from a calibrated clock source (use the PPS pulse to discipline an oscillator). I was too lazy to build such a circuit and observe the effects. Some uBlox timing GPSes let you output a suitable signal (you can adjust the frequency in software), but the adjustability range when I last checked wasn't enough to make it work for the Beaglebone. Once you have the MCU running at a well-defined and stable frequency, I think it's possible to give Linux extremely accurate data, or at least know which clock cycle was the start of a second. I don't actually know how much any of this will help, and it's honestly not within my capability to measure.
For me, the outcome of my GPS clock project is that I never have to set my clock, and it looks really good. Running "chronyc sources" is always pleasurable as well, but I probably wouldn't notice a clock that's 20ms off unless I was comparing it against WWVB or something.
A WWVB receiver is on my todo list. I did buy a cheap module that they use in clocks, and I get no signal whatsoever (New York City). WWV (shortwave) I can hear sporadically. That is basically a result of the modules using the legacy WWVB signal, and not the newer phase-modulation signal, which is easier to receive at low signal strengths.
That's a useful feature of the BeagleBone. When I originally built my clock with a Raspberry Pi I used a GPS with a PPS line and hooked that into one of the Pi's GPIO lines. I could get the Pi's clock synced to the GPS very accurately (easily sub-ms) with that (using chrony), but the problem became then updating the display (similar to your setup, a 7-segment MAX7219-driven display). I started looking into writing a kernel module to update the display when the PPS-driven interrupt happened. I think, from reading the kernel documentation, there may even be some support for firing off events when the PPS occurs, but I realized it might be "easier" at that point to move to an arduino-ish solution.
I agree that I wouldn't likely notice a ~20ms jitter on the display, but I'm kinda of having fun figuring out how to make it as accurate as possible.
I measured how long it took on average to write out the display, and I set my program to write out the display at that time. The values are in the source code in that repo, but if it was 1ms, at 23:59:59.999, I'd wake up and write out 00:00:00. (Scheduling jitter could be a problem, but comparing when my program starts writing data to the actual PPS signal with an oscilloscope, it seemed amazingly consistent to me. I don't have my data around to really confirm that, though.)
I have that video linked to Github and within about ~10ms, you can tell that things are working really well when the audio and visuals line up. (I say ~10ms, because there is a 2ms propagation delay between the radio station I'm receiving and the clock display changing, and it looks perfect to the eye. If it's a little longer than that, you'll notice -- WWV with ~8ms propagation delay makes you question whether or not the display is updating correctly. It is hard to discern 8ms, of course, but I think it's noticeable if you're looking for it. This, BTW, is why I hate 60Hz monitors. Your brain definitely notices the 16ms it takes from pressing a key to it showing up on the screen. We get used to it, but why tolerate failure when you can make it perfect?)
If you're looking for a relatively low-cost option for "always correct" wall clocks, Primex radio clocks are popular enough that they are very cheap used on eBay, and if you live in a city there is a decent chance there is a GPS-disciplined Primex transmitter within receiving range (hospitals, schools commonly have them, you can check with an SDR). Or for a standalone solution, the sadly discontinued Primex SNS series of digital and analog wall clocks are just WiFi NTP clients. Supply is becoming limited since they were discontinued but they still run fairly cheap used. Initial setup (WiFi credentials, NTP server, etc) is slightly eccentric but can be done with tools available online.
The analog wall clocks are battery powered and with the default config of re-sync every 3 days, the batteries last years.
This is one of the reasons for OpenNTPD's existence. They wanted something that was simple and a ~50ms deviation was part of the design goals/trade off for this.
Man, tough crowd here. And someone who's been following precise timekeeping tech for 25 years now.. this is really cool! I think it's great some smart engineers have built a new reasonably compact and low cost very precise bit of timekeeping hardware. Thank you for sharing it open source!
I am torn. I absolutely love the technical blog post and the effort that went into open sourcing everything needed to build these devices, but, you know, Facebook. Compliments and respect to the engineers involved for really well done engineering.
Objectively speaking, Facebook has done both great things (e.g. from an engineering/tech perspective) and evil things (we all know). It's good that we honor the good too when we are flaming them for the bad.
Amazing people at Facebook make amazing things, but it does seem like only the worst of them (FB, Google) have the cash to burn to hire teams of engineers for projects outside of ones that directly contribute to generating revenue.
They've built and earned their (negative) reputation. This is cool, but it's like Philip Morris Tobacco Company coming out with a useful home appliance. Uhh....thanks?
This is a pretty neat thing and kudos for them open sourcing it. Typically, you'd have to hand over $5K - $25K for this card! This can hopefully make them available at a lower price.
I built something similar (although I didn't use a CSAC[1] since it was too expensive but I did have a nice OXCO which is pretty stable. I use a Raspberry Pi to distribute plain old TOD via NTP to my local network, but the 10 MHz output I send to my bench where I can use it for radio projects. It is actually a better clock source (frequency stability wise) than the precision time source in my HP spectrum analyzer but it has worse phase noise (jitter) because it does get corrected by up to +/- 20nS periodically.
Really nice of them to release this. However, unless I'm missing something, the HDL for the FPGA on the Time Card is not part of the release. It looks like it uses a proprietary IP core from NetTimeLogic
> The offset is practically 0 ranging within nanoseconds range.
Uhm, no, the graph clearly shows that the offset is varying between 0-2 microseconds, which is 0 - 2,000 nanoseconds.
2,000 nanoseconds is not "practically zero". PTP precision is supposed to be in the 10ns range.
Also, how did they get 40us precision on (what I assume to be) a normal kernel on a normal x86 box? I would have assumed that in a datacenter environment, with traffic levels being random, the jitter introduced in the kernel networking stack alone would be in the hundreds of microseconds at least.
Addenda: Intel networking chips, so even in AMD or ARM systems this actually works (assuming that you have the correct OS and drivers/modules/support software).
Also, this isn't exclusive to Intel: most modern (server) NICs at least support TCXO-based timings, and some built for precision uses OCXO for even better precision.
I am very thankful to all of you putting all these questions and answers. We have a bi-weekly meeting under the open compute project and you can find previous talks there too. www.ocptap.com
A few years back I was interested in microsecond-precise timekeeping.
My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive.
And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch.
I wonder how Facebook are connecting their precise new time servers to their regular servers?
There is a big push in the Industrial automation world to get TSN/PTP deployed more widely and perhaps universally. The idea is to eliminate the dichotomy of real time vs non real time Ethernet networks in control systems from the plant network down to the machine level.
This way you can send EtherCAT frames at equidistant intervals to servo drives in the microsecond range while casually streaming SPC data or Netflix over the same wire.
This is also in conjunction with the newer single pair Ethernet tech that the Automotive world is also interested in.
> In addition to the more computer-oriented two and four-pair variants, the 100BASE-T1 and 1000BASE-T1 single-pair Ethernet PHYs are intended for automotive applications[17] or as optional data channels in other interconnect applications.[18] The single pair operates at full duplex and has a maximum reach of 15 m or 49 ft (100BASE-T1, 1000BASE-T1 link segment type A) or up to 40 m or 130 ft (1000BASE-T1 link segment type B) with up to four in-line connectors. Both PHYs require a balanced twisted pair with an impedance of 100 Ω. The cable must be capable of transmitting 600 MHz for 1000BASE-T1 and 66 MHz for 100BASE-T1.
> Similar to PoE, Power over Data Lines (PoDL) can provide up to 50 W to a device.[19]
> The IEEE 802.3bu-2016[12] amendment introduced single-pair Power over Data Lines (PoDL) for the single-pair Ethernet standards 100BASE-T1 and 1000BASE-T1 intended for automotive and industrial applications.[13] On the two-pair or four-pair standards, power is transmitted only between pairs, so that within each pair there is no voltage present other than that representing the transmitted data. With single-pair Ethernet, power is transmitted in parallel to the data. PoDL defines 10 power classes, ranging from .5 to 50 W (at PD).
100BASE-T1 is IEEE 802.3bw-2015, 1000BASE-T1 is IEEE 802.3bp-2016. 2.5 Gb/s, 5 Gb/s, and 10 Gb/s over a single pair is 802.3ch-2020: the focus of these is in the embedded automotive space.
Most (if not all) our switches in the datacenter are capable to perform as transparent clocks. With transparent clocks you can keep the time sync error down to less than 5ns per hop
Do the transparent clocks in your datacenter support unicast PTP?
There is a possibility to use PTP as a transport for NTP to take advantage of PTP-specific hardware timestamping. It could also process the correction field, but it seems the switches typically don't support unicast PTP.
The large asymmetry and banding of NTP in the test with Calnex Sentinel suggests it doesn't support the interleaved mode. NTP with hardware timestamping should normally be much more stable and symmetric, closer to PTP.
The unicast PTP support can be limited to boundary clocks and the enterprise profile doesn't require transparent clocks to support the unicast mode.
Also, there are different types of PTP transparent clocks. They can either be end-to-end or peer-to-peer, and either one-step or two-step clocks. To be useful for NTP, I think it would need to be an end-to-end transparent clock and ideally it would be a one-step clock to avoid dealing with with the follow-up messages.
Even the retail price of 1 switch based on their "wedge" design is less than $10k for 32 100gbps ports. Merchant silicon like the Broadcom 56000-series switch ASICs support PTP.
There are consensus protocols that allow a grace period for clock skew, and the narrower you can constrain the clock skew, the more transactions per second you can retire, improving both scalability and latency.
I work in trading, but mostly on the asian markets, so not sure about mifid. Usually the exchanges seem to give milli to second precision (which is very annoying: if Thailand says it's second-precision to timestamp their order executions but Taiwan says it's milli, our system has to use the most precise when ordering them and we get questions from our own multi market clients about why all these executions are seemingly simultaneous or why our system source marked the request at .480 while the exchange replied with 0.0 for the execution.
But, by no mean, does it mean we use special timing - but now that I say that, I realize I have no clue where our time signal comes from, clearly it wouldn't be the internet because our trading servers aren't exactly directly connected there, we must have an internal ntp sever o_O
Not all exchanges are like this. The JPX exchange in Tokyo provides PTP over dedicated cross-connects, and is super accurate, within +/-25ns. It’s pretty neat seeing a PTP clock from Seiko.
There are plenty of practical applications for it.
Imagine super high speed, high precision robotics, maybe a synthetic fiber layup machine or an exotic milling machine, that moves at 10m/s. If you have 1us of precision and accuracy then you can send movement commands that are precise down to 10um.
In these cases mostly not, they require low jitter references and a good counter.
An example that does - I work for IceCube, a big neutrino detector at the South Pole. The detector is essentially a time of flight system that detects emitted light as it propagates through the ice when a neutrino interaction happens. We need good time distribution in the system so we can marry up hits from the 5000 odd sensors in the ice (and successive hits may be nanoseconds apart), and we need good absolute precision because we collaborate with other observatories.
Also pretty much all radio telescopes that do long baseline interferometry need excellent absolute timing. The South Pole Telescope has (I think) the most accurate clock on station, a hydrogen maser-based system. We use GPS.
> While building our Time Appliance, we also invented a Time Card, a PCIe card that can turn any commodity server into a time appliance.
Invented? These cards have been around. e.g. Meinberg makes a variety of expansion cards which use various time keeping sources such as WWVB, IRIG-B, GNSS, etc.
Please excuse the language “invented”. We simply meant “came up with a form factor” of having the MAC+GNSS present via PCIe bus as a POSIX clock similar to a NIC with hardware timestamping. We are humble to all the awesome products that are out there and we just wanted to present an open source version of a simplified approach to allow others to build on it.
Isn't this a Symmetricom card? Being open source is the killer feature here. There are a number of companies that make PCIe timing cards, but they are all proprietary and go out of support way too quickly.
I read it that they've invented a product called time card, rather than the concept.
Its an accurate and stable clock source. which is an important distinction. The Meinberg PCI cards appear not to have a local clock source thats well disciplined (ie a temperature compensated oscillator)
Having an accurate time source that is updated once every second (typically) with a 1pps GPS is grand for most things. But if you need a stable clock as well, then you want a tiny atomic clock onboard as well. This is so you can run a clock source for an entire datacenter, or local section.
Those have the external source but don't seem to have the internal reference (Rubidium, Cesium, etc) for holdover.
FB seems to be claiming to have invented a PCI card that makes a commodity server into a Stratum-1 source. Meinburg sells that too, but in proprietary appliance form.
Again, invented might be too strong. This is an abstraction of what you need minimally to build a time server on an open source basis. We appreciate and are humble to other products in the market
The availability of a much cheaper product than the appliances will likely create a bit of stir with competitors. The article mentions better management functions too. So perhaps invented is too strong, but it's more notable than your follow up comments suggest.
What sounds like the most reasonable strategy to me is to have a single atomic clock in every datacenter, and then some basic hardware in every server to communicate with it.
Which is exactly how the telephone industry has done it for decades, it's called a "building-integrated timing supply", or BITS clock. Typically this is a GPS-referenced clock with a local quartz or rubidium holdover oscillator, and output cards with dozens or hundreds of outputs serving various machines in the office.
CDMA needs precise sync for soft-handoff, but that's tangential to the rest. SONET explains it better, but even SONET is decades newer than the original requirements.
The T1 was introduced in 1962, having been developed in the 50s. T1 is self-clocking, in that the receiver recovers sync from the line (and there are mechanisms to guarantee enough ones-density to keep the clock recovery circuit working), so a point-to-point T1 circuit with analog on either end has no need for external timing -- one end can just free-run and the whole thing is fine.
But when you start connecting T1 (DS1)s together, moving DS0 signals between them ("time-slot interchange"), or bundling them into higher-rate signals (DS3/T3 and then up into SONET), it's utter chaos unless they're all synchronized.
There's considerable detail in the BSTJ archives, I could dig up some links later.
Not in telecommunications, but I do know that there are synchronous comm networks that don’t require any kind of sentinel / delimiter because there is a guarantee that the voltage will agree.
My signals background mostly comes out of music, so somebody else probably has a better answer. But any time you need to convert between digital and analog signals relatively seamlessly, ie, introducing minimal processing overhead, you’re going to run into these kinds of issues. In that regard, you can think of telecoms as a CPU, and just like in a computer, you need a system clock that the entire architecture agrees on.
First likely application that comes to my mind is VoIP.
It is one of those problems that is really easy to ignore, until you can't. I was using a cheapo 8051 based logic probe the other day and had a lot of fun dealing with not only jitter of two different oscillators (probe + device under test) - but differences in response time between probes (within the same sample period). If you ever find yourself wondering much influence your MCU's 4 clock cycle per instruction architecture has on the waveforms you're looking at... you're about to have a very bad time.
It's not about accuracy (nobody really cares if the whole network is running fast or slow compared to some other clock) so much as synchrony (the whole network needs to be running from the same clock).
Asynchronous networks can't guarantee latency or jitter performance, and they require buffers. Synchronous networks (remember, this is voice-grade stuff) can deliver bits out at the same rate as bits in, anywhere in the network, without dropped frames or stuffing or anything obnoxious like that.
Analogy time: If all you're familiar with is packet-switched networks, sure, let's call them trucks. A truck gets filled with payload, it leaves, and sometime later, it arrives at its destination. Packets get reordered, and it's someone else's problem to deal with that. A great many shipping-and-receiving docks contain a great deal of complexity, to check bills-of-lading against what was ordered, with warehouses to smooth over the difference. It works today because computers are powerful, and buffer bloat is still a plague.
Implementing VoIP with 1960s technology was... not practical.
So, synchronous networks don't require any of that complexity. Nothing's ever out of order, there's never a bit arriving too soon or too late. You don't even handle them as packets, they're literally just streams of bits, coming out of an ADC here, beating through the network in synch, and being shoved into a DAC over there. Rather than a truck, imagine a bunch of slow conveyor belts feeding a faster-moving belt with a spinning turnstile or little paddle controlling what goes onto it -- for this microsecond, input 1 can emit a bit onto the belt, then a microsecond later, it's input 2 and so on. So the belt contains an ordered stream of bits, and at the other end there's another paddle spinning in sync with the first, demultiplexing the bits back out to their sub-rate circuits.
This can be implemented with discrete transistors. There's no need for buffers or packet reordering or any such nonsense. And definitely no repeated frames. There are no hiccups, no dropped frames, no reassembly or reordering. And definitely no repeated frames. End-to-end latency is the speed of light plus one half a frame time, on average.
And if you can make the whole continent beat in sync, you can enjoy this level of service and predictability even on long-distance calls.
Initially (in the 1960s, as time-domain multiplexing was deployed) this was done by having one master oscillator in Kansas City (the approximate geographic center of the continental US), and fanning-out its timing through an enormous branching-tree structure of amplifiers called Synchronization Distribution Expanders. This performed beautifully but it was a logistical nightmare, any network disruption could cause problems, you needed local oscillators for holdover anyway, etc.
As GPS was made available for civilian applications, it made sense to convert to using Schriever AFB as the master clock, use the space segment as the distribution network, and simply give each office an antenna to tap into it. The implementation is a bit tricky because of doppler shift and stuff, but GPS receivers abstract all that away, and can hand you a timepulse which is synchronized to GPS system time. (Then the BITS rack converts that to the useful frequencies and distributes it within the office.)
Nothing off the top of my head that's succinct and readable.
You'd do well to paw through the Bell Systems Technical Journal -- there's some amazing stuff in there like the origins of Unix and C [1], or Shannon's information theorem [2] -- but some details of the network itself are either hard to search for, or might not be there at all.
Most of the old phreak philes are focused on the billing and routing aspects of the network, not so much on transport. I remember a few test equipment manuals had good introductions to the structure of a T1 frame, for instance, but perhaps not an exhaustive treatment of how that came to be.
This is an interesting question. It must be out there, mustn't it?
A very long time ago, HP made a GPS time card just like this, encased in glass and filled with an inert gas such that if the card was tampered with, it was rendered useless.
This prevented someone from faking the time signal.
Buddy of mine owns timestamp.com and we were building a service where you could hash/timestamp emails (we would take over the MX for a domain and become a forwarding service). All of this was long before we had blockchains.
Sadly, it never really took off so we abandoned the idea (what is up there now is currently another half attempt at the idea).
Others have tried, but it isn't really a huge demand despite the apparent need.
I would caution people against using the Mellanox/NVIDIA network interface cards in anything they care about. The driver license is proprietary and you have to build it for your kernel as a DKMS module in a semi manual process. Makes future system updates a real bother. I realize this probably doesn't apply so much to FB internally since they build and maintain their own distro in house.
Stuff that's just a couple of years old won't even build the driver for current debian bullseye. Have to run buster? No thanks.
As with so many other things NVIDIA the root cause of this is their absurd licensing approach to open source software and drivers for their hardware.
Where did you get that from? The drivers for NVIDIA networking cards are in the upstream kernel. You have the option of using newer versions from DKMS if the built-in version for your kernel is too old.
How about the x2 or Alveo series from Xilinx? It is basically the original definer of smartnic.
It comes from solarflare who have a long pedigree of low-latency smartnics. They used to supply Cloudflare, and also supply like 50% of fintechs/financial markets.
You can also just use openonload to accelerate your programs. In this case just doing straight linux socket programming, which can be accelerated without dpdk. Or just use the open source linux net driver if necessary.
I recommend Intel for 100GbE stuff. Intel's heritage of proper open source drivers (written by their own staff, in house) for FreeBSD and Linux goes back really far.
You could look at the man pages for their very expensive 10GbE server NICs in 2003-2004 and see the @intel.com email addresess of the persons who wrote them.
Seiko offers a GPS clock in a conventional-home friendly form factor.
First introduced in 2013, the Seiko Space Link GPS clock has proven to be a global success. It is the world's first clock for home or office use that receives a time signal from a GPS satellite, thus delivering atomic clock precision. It measures time to within one second every 100,000 years. Equally importantly, it can be used anywhere in the world and can be installed in almost any situation, so long as it is around 15 metres or less from the open sky. The technology is beautifully practical. 4 times every day, the clock receives a time signal from the GPS satellite network and, when conditions are ideal, adjusts automatically in around 10 seconds of receiving the signal. Between these signal receptions, a high accuracy quartz movement keeps the clock on time.
Wherever you may need it, Seiko Space Link is there, silently marking the passage of time with an elegance and precision that is redefining the world’s perception of what a clock can be. A new standard of excellence has arrived.
This is a standard "atomic clock", that its not actually uses radioactive decay to accurately measure the passage of time, but receives GPS signals. The innovation in the post is using real "miniature atomic clock" that actually uses caesium 133.
To improve the performance of their distributed systems by imposing stronger assumptions than full asynchronicity on their environment, I expect. (See also Google’s Spanner.)
Sure, I didn’t intend to make any deep point here; I don’t think I have any deep points to make in this particular area, honestly :) I just wanted to drop a set of easily searchable terms for motivation that might not have been made explicit in your writeup (probably because it’s quite tangential to it—metrology and distributed systems are both fascinating but don’t really share a lot of ideas).
There are probably a few reasons. One that I can imagine is that they use a distributed database somewhere that benefits from reliably tight wall clocks (in the style of Cloud Spanner or Cockroach).
Cockroach is much less dependent on highly accurate clocks. That was basically the core goal. Spanner like, without expensive clock infrastructure. Though it sounds like CockroachDB does better with at least reasonably well synced clocks.
True, I'm not disagreeing with you. However, it's worth nothing that Cockroach DB's consistency model assumes consensus in 180ms, and Spanner assumes similar SLOs at 6ms. Arguably, one could say that the reason why an open-source spanner-like DB doesn't exist in the marketplace with comparable performance is because these types of cards are proprietary and close-source (i.e Google's TrueTime). To your point, Cockroach is much less dependent on highly accurate clocks, but if these PCI-e cards became a common hardware commodity Cockroach can become even faster.
Now imagine a world where distributed DB technologies get so good that the trade-offs between consistency and availability is enough to warrant the use of a distributed DB for mid-sized projects. Right now, these DBs are seen as muscle cars with high cost and high maintenance only for BIG projects. A card like this might be the game-changer.
From the article: "More accurate time keeping enables more advanced infrastructure management across our data centers, as well as faster performance of distributed databases."
That seems rather ambiguous and doesn't really explain why having more time precision results in faster performance or 'advanced infrastructure management'.
I can understand why a stock broker or trading firm requires PTP to enable precise date-stamps for auditing/validating trades.
I don't see how having a time granularity on the order of picoseconds is needed for a data center.
In distributed databases that offer transaction semantics, they need timestamps to order transactions that take place. A tighter synchronization of clocks mean they can execute transactions faster because they can reduce the amount of time they wait (based on the potential clock drift between machines in the data center)
If you know that your clocks are exactly the same in New York and Japan you can assume that if an account creation timestamp in Japan is before the account creation time in New York for the same username that the account should go to the Japan user and the New York user will be told that he cannot use that username. Previously the way to handle that problem was to have all account creation be done on one server which will block the table while it is doing writes. That doesn't scale when you need to insert millions of rows a second.
When you have collisions, the timestamp might be the thing that decides which update came first. The higher the precision of the time stamp, the faster you can run stuff.
If you want the go under the nanosecond there is the White Rabbit [1][2] project. The CERN uses it to control the magnets in the loop of a particule accelator. It is also based on NTP, Ethernet and FPGA.
Facebook isn't a good fit for me, but their work on open hardware for data centers has occasionally looked appealing.
I figure units doing open hardware could be like when the old SNL sketch about AT&T -- having the slogan "We don't care. We don't have to. We're the phone company." -- was of the same company that hosted Bell Labs.
These timing cards are low volume products so they have a bad habit of going out of support after a few years and then the kernel level drivers don't get updated and there is hardly any community. Having open hardware that multiple vendors can support makes them easier to get added to the mainline kernel.
Thats exactly the point for this project. Also, I am hoping once we release the FPGA HDL files, the community will start adding more features like IRIG-B output, etc...
Am I missing something? The FB blog proudly states how they improved accuracy "10 milliseconds to 100 microseconds".
Surely that's a lot of effort that anybody with in a datacentres with decent connectivity could replicate with a Chrony instance ?
100 microseconds is 0.0001 seconds, right ?
Seems to be something Chrony can do with its eyes closed:
System time : 0.000003297 seconds slow of NTP time
Last offset : +0.000000549 seconds
There are lots of great NTP servers out there hosted by trusted parties such as national time labs. Most people don't need fancy GPS hardware (and the expense of paying someone for roof access to put your antenna !). If you've got good software and good connectivity you can replicate 99.999% of the same thing.
This seems like a cool project. I didn't see a price estimate for this PCIe card.
However, isn't it smarter to try to build an infrastructure/architecture that can deal with somewhat less precise time information (milliseconds instead of nanoseconds)?
I'm still unclear why they need the MAC on top of the GNSS reception. I can understand if it's just to provide holdover in event of GNSS signal loss, but they seem to be saying that it provides better precision when you've got both.
For anyone wondering, MAC = Miniature Atomic Clock. From the vendor product page:
> Microchip’s next-generation MAC-SA5X miniaturized rubidium atomic clock produces a stable time and frequency reference that maintains a high degree of synchronization to a reference clock, such as a GNSS-derived signal, despite static g-forces or other factors. Its combination of low monthly drift rate, short-term stability and stability during temperature changes allow the device to maintain precise frequency and timing requirements during extended periods of holdover during GNSS outages or for applications where large rack-mount clocks are not possible.
Generally: if accurate positioning, navigation, and timing (PNT)—especially timing—is important in your infrastructure, then you need to plan for GNSS outages.
> In the event of the GNSS signal loss, we need to make sure the time drift (aka holdover) of the atomic-backed Time Card stays within 1 microsecond per 24 hours. Here is a graph showing the holdover of the atomic clock (SA.53s) over a 24-hour interval. As you can see, the PPS drift stays within 300 nanoseconds, which is within the atomic clock spec.
That’s a good question.
GNSS gives you a 1 pulse per second signal. It is often with 20ns of accuracy. Now you need to interpolate nanoseconds between consecutive 1pps pulses. This is where a high stability oscillator is needed. You can certainly use an OCXO for the fraction of the cost of a MAC. We went for the MAC because of the importance of the role that the device plays in our datacenter
Although clock holdover might seem like a pointless feature - GPS outages are very unusual* - what holdover is mostly for is giving you time to respond if someone clumsily knocks your antenna over, or crushes your coax cable, or it gets damage that makes it fill with water, or something like that.
A lot of the demand for high-precision clocks is for cell phone base stations, where there's no guarantee there'll be someone on hand to make repairs promptly.
In downtown mountain view I saw brief GPS outages a couple times a week, presumably due to people driving around with GPS jammers (e.g. to knock out GPS tracking of their own cars).
GPS spoofing is also a thing. Here's a paper where they are able to spoof time while letting the receiver think it hasn't moved (much, within error bars): https://ieeexplore.ieee.org/document/6451170
That paper is in context of power grid equipment, but the GPS attack generalizes.
The GPS can easily wobble by 50ns back and forth as the constellation changes. That's a lot! And, it is not random on a short time scale.
Folks often think "Oh, +/- 50ns, 20ns RMS, easy to filter...", but that's totally wrong.
The GPS will report -30ns from stable for minutes on end, then slew to +10ns, then -5ns, etc. Any high-precision oscillator (such as for radar) that's being jerked around like that isn't going to be as stable as high performance needs.
Even for just handoff of handsets at 2.2-2.3GHz, having the radio network (aka cell towers) all locked to an oven-controlled oscillator that was aligned-to, but far smoother-than, GPS, made a huge difference.
Now, improvements to GPS/GNSS that track 12 satellites instead of 6, and across multiple constellations, can result in more stable radio-based time. But then you get into urban canyons, and can only see 5 instead of 12, and you're right back into the jumpy situation.
GNSS PPS is more jittery but does not drift. The MAC has 1000x less jitter but drifts. You can cleverly combine them with statistics and magic to get the best of both worlds.
I've implemented this (using GPS PPS to discipline a clock) using regular old PI control loops. I just make the loop damped enough to filter out the GPS PPS jitter and you're good.
From what I remember of textbook discussions of atomic clocks, stabilizing drifting oscillators is also in part how normal, old (50+ years) atomic clocks work as well, isn’t it? You have a fiddly and unreasonably high-frequency reference, you slave a relatively garden-variety standard like an environmentally controlled piezoelectric crystal to it, and then feed your periodic signals, counters, etc. off that.
Yes. Surprisingly, atomic clocks have high jitter compared to ovenized quartz oscillators. Every atomic clock package is outputting its signal from a quartz oscillator that is disciplined by the atomic oscillator.
You use a jittery GPS PPS signal to discipline a highly stable primary clock through a very "slow" filter. The GPS PPS might have an RMS period jitter of say 10-20 nanoseconds, but the drift will be 0.
Then, you can use that disciplined primary clock to provide actual time for timestamping, etc.
I think they mean better precision during holdover because they model any inaccuracy of the rubidium clock while GNSS is up in order to compensate the local clock when GNSS isn't working.
Off topic: I'm reminded of a very small unit of time that was devised to be a least common multiple of various division of time like 24fps, 30fps, 44.1kHz, and a ton of others in video and audio. Does anyone remember what this unit was?
I would imagine it's archived because there's nothing else to really be done. It's basically a blog post. The header can apparently be replaced with three lines.
> Now, as long as printing the PCB and soldering tiny components does not sound scary, anyone can build their own Time Card for a fraction of the cost of a regular time appliance.
Doesn't sound scary, but not a lot of research labs and data centers are going to want to build their expansion cards from parts, even given free access to instructions. I understand if there isn't enough of a market for this to make it a viable commercial product, but when you have more money than God anyway, what's the reason not to make a few extra and donate or sell them to the small number of users who can use it?
Of course we are not going to build the card in the lab with assembling SMD parts. This was more a way to explain that building such a card should not be out of access for anyone
You are right. This sentence was not supposed to be a literal description of what is the best way to build the card. It was more referring to the fact the building this card is not that hard since the main goal for this project was to liberate the time appliances from being a closed source box.
> […] we are currently working with several suppliers and will have their contact info soon available to allow you to puchase an out-of-the-box ready Time Card.
For anyone else wondering how their clients verify the authenticity of timestamps; It seems they use the NTP server software Chrony, which has support for NTS (Network Time Security, RFC 8915).
Are there further details available about the MAC?
Growing up in Boulder, THE atomic clock (NIST) was touted as a modern marvel (circa early 90’s childhood), it’s incredible to think that similar performance is now available on a pci-e card! Any relation to DARPA’s “making progress” announcement in August 2018 [1]?
How resistant is this to a GPS attack? It has an atomic clock, so it should be producing good data for days while sending alarms in the event the GPS data is bogus. You don't want to create a situation where someone aims a small dish at the roof of your data center and takes down the networking.
I have been working on a version with two GNSS receivers L1/L2 and L1/L5. The SA53 receives two PPS inputs and that allows you to detect Spoofing and Jamming.
Again, I like to emphasize, this is not a done work. We shared whatever we had and we would love to have the community to contribute and help making it better. I love to have as many applications as possible for this Time Card.
For example, I am looking for someone willing to port/write a Windows driver for the time card.
With regards to the VHDL code; we are working with NetTimeLogic on a version without the need of any of theirs proprietary IPs. Meanwhile, we provided you with whatever we have and we look forward to working with you.
Please feel free reach out to me if you like to collaborate on this project.
> The new service, built in-house and later open-sourced, was more scalable and improved the accuracy of timekeeping in the Facebook infrastructure from 10 milliseconds to 100 microseconds.
Interesting.. the infrastructure must therefore have a diameter of at most 30 km, thanks to the speed of light
How well does transaction rate increase with more precise time? Seems like it would be a sort of O(log n) progression. Though could very well just be linear as well.
I don't see a big deal here.
You are using a very expensive free-running time-base (MAC - Rb clock) to implement the holdover. These MACs are really expensive, and basically unobtanium if you are not a big corp.
You should have done it with a DOCXO and some clever training software to correct the DOCXO vs temperature and aging when GPS tracking is lost.
Of course they come out with this now, after I've got a few antennas up on the roof, half a dozen GPS-disciplined OCXOs providing 10 MHz and PPS signals throughout the house, which are being fed into SolarFlare PTP NICs, and sync'd throughout the network using PTP.
Apparently my workstations' offsets (RMS) are currently 17 and 21 nanoseconds, though, so I suppose there's still room for improvement.
Mobile basebands already get the local time and UTC offset when connecting to a tower, there's no need to power up the GPS antenna and receiver separately.
Mostly because 1) your phone is getting time from the tower anyways (they have to be closely sync'd) and 2) GPS doesn't always work very well -- especially indoors.
Slaving your clocks to GPS seems like it just gives the USAF space command a way to, eventually, wreck your operations. If you don’t have requirements for accuracy referenced to civil time (such as mifid 2) why wouldn’t you just go with the atomic clocks alone, with gps as an advisory-only monitoring reference?
Probably because an ESP8266 isn't anywhere close enough to "real-time" to be of any benefit.
Why bother getting a nanosecond-level accurate time signal if you're going to feed it to an ESP8266 to distribute over Wi-Fi (or, in other words, reducing your accuracy to ~10 milliseconds or so)?
I didn’t want to click this so I assume others here won’t either. Here’s the top points they highlight themselves:
- Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure.
- To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance.
- With the help of the OCP community, we established the Open Compute Time Appliance Project [1] and open-sourced every aspect of the Open Time Server.[2]
Yes, I appreciate their engineering work. I did have to click the link to post my comment. I posted it for other people’s benefit because I understand the concerns regarding their privacy record.
I've seen plenty of projects described as open source when the firmware is open source and the hardware is closed but this is the first time I've seen one where the schematics and board layout are open but the firmware is closed. Note that the hardware itself is quite simple, the smart stuff happens in three modules: GNSS receiver, precision clock and FPGA. To me, the contents of the FPGA source code are the only interesting part of this project. Additionally, you won't be able to meaningfully modify or reuse this project without editing the source code.
The hardware module these FPGA binaries seem to be compiled for is described as an AC7100B, the source given for these is a defunct ebay link [3] [4] :-O This project uses two and half grand's worth of atomic clock and the heart of it runs on a module that fell off the back of a lorry!?
[1] https://github.com/opencomputeproject/Time-Appliance-Project... [2] https://www.nettimelogic.com/clock-products.php [3] https://github.com/opencomputeproject/Time-Appliance-Project... [4] https://www.ebay.com/itm/XINLINX-A7-FPGA-Development-board-A...
Edit: I realise this comes across as quite negative. This looks like a neat project, one I would actually use, but only if it were meaningfully open source.