CDC methodologies for success
Dave Wallace, Chief Scientist, Blue Pearl Software
The designs of both Application-Specific Integrated Circuits (ASICs) and FPGAs use more and more clocks these days. Ninety percent of ASICs have multiple clock domains and for FPGAs it’s over 85%. That means it becomes more and more important to deal with the interactions between different clocks and the clock domains they’re part of. This can end up producing bugs that can escape into production – and often do if you don’t wind up checking for them. So let’s see what’s going on.
There are basically two different types of clock relationships. Clocks can be synchronous to each other, for example, in the upper part of figure 1. The figure shows them connected by a buffer, but it could also be an inverter, a clock divider, or other similar logic. It’s possible the clocks could be running at slightly different rates. The key is, do you know the phase relationship between the different clocks? Do you know how a clock edge on the first side interacts with a clock edge on the second side? If so, you can use static timing analysis to verify your timing and everything’s good. Groups of synchronous clocks will define a clock domain where each clock in the domain is synchronous to every other clock.
That’s the ideal state but in reality we often have multiple clocks and we may not know where they come from, or we may know that they’re completely independent. That’s the asynchronous relationship in the lower part of figure 1. In that case, you can’t generally use static timing analysis or timing simulation to guarantee the timing correctness of the paths between asynchronous clocks. Sometimes it is a design team judgment call. There may be clocks that are technically synchronous but the closest approach of the clock edges is too short or too uncertain to be able to use static timing analysis. The design team in that case may choose to treat clocks as asynchronous even if technically they could be synchronous.
This is where Clock Domain Crossing (CDC) analysis comes in. It’s any time you have data that’s sourced in one clock domain and received by another. There may be a logic cloud between them, there may not, but if you can’t determine the relationship between the clocks you can’t guarantee that setup and hold time violations don’t occur, and that can cause various kinds of problems. Primarily that would be metastability but it can also lead to data corruption where you have data values being out of sync and this is something that needs to be addressed.
Metastability is hard to find
Metastability is the major problem that we’re trying to address in CDC analysis. If you can’t guarantee that the data is arriving outside of that setup and hold window, there’s a timing within that window where if the data comes in at just the wrong time relative to the clock, the output of the flop may actually go metastable. That is, it may assume an intermediate voltage between a logic zero and a logic one, and it may stay there for a certain period of time.
A metaphor we often use is that it’s a little bit like flipping a coin. Most of the time you flip a coin it comes up clearly heads or clearly tails but every so often it comes down on edge or nearly on edge. Most of those cases will resolve fairly quickly into a clear heads or tails but sometimes it can dangle there if you hit just the right conditions. You really don’t want this kind of metastability propagating through your circuit because you can’t really guarantee the behavior of the gates in your circuit if the inputs are going metastable. Industry reports show that clocking and CDC errors are the number two cause of respins in designs so this is a real problem and it’s one that’s very hard to detect.
First of all, for a lot of FPGA developers the CDC analysis may not even be present in your FPGA vendor offering toolsets or it may only run during place and route at the very end of the design. It depends on the precise timing of the receiving clock versus the data and for some cases like Register-Transfer Level (RTL) simulation you don’t even have that. You only have cycle accurate timing. But even timing simulation doesn’t know the phase relationship between asynchronous clocks and you can’t time the paths without some sort of bounds on that phase relationship.
Enter CDC analysis
CDC analysis generally uses a different default assumption than other tools in terms of whether clocks are synchronous or asynchronous if unspecified. A lot of the existing timing representations such as Synopsys Design Constraints (SDC) assume synchronous clocks by default because they come out of a static timing analysis or synthesis approach where they initially assume all paths can be timed. CDC analysis on the other hand is focused on what’s going on with the asynchronous clocks – are the paths between asynchronous clocks properly synchronized or not? You don’t want to have the clocks synchronous by default for a CDC tool because that would shut the tool up about things that could turn out to be a real problem. In either type of analysis, you don’t want a tool silently suppressing possible errors.
CDC analysis also looks at the external environment. It’s not just paths within the design or on chip that you have to worry about. It’s the interactions with the external environment where you may have paths coming into the inputs of your design that originate on some clock, and paths going out that are being received on potentially some other clock. Some of these clocks may be virtual clocks because they don’t appear in your design and they originate from some clock that’s completely off chip and that you never see. So you have to deal with making sure that you can specify what clocks external I/Os are associated with. In dealing with external resets you need to identify when those resets originate or if there are any constraints on those resets.
Setting up for CDC analysis
The first step in any CDC analysis is to define associated clocks for the I/Os as much as possible so that you can tell the tool as much as you can about what clock domains the signals coming in and going out are related to and which clocks are synchronous to each other. That’s where a clock interaction diagram can help you group synchronous clocks into domains.
We see in figure 3 that each of the ovals is a clock and the boxes indicate domains where the clocks have a synchronous relationship to each other. Each arc is potentially an interaction between two clocks, indicating one or more paths originating from one clock and being received by another. The annotations on the arcs tell us how many of the paths are already synchronized or not as well as some other information.
For example, look at the arcs showing the interaction between the top clock, clk_i, and lwb_clk_i to the lower right. Here, we’ve got nine interaction paths coming in to lwb_clk_i, all of which are unsynchronized, and 72 paths going back, all but one of which are unsynchronized. That suggests that whoever designed this logic either didn’t know or care about synchronization or, maybe, was thinking of these two clocks as being synchronous to each other so they didn’t need to worry about synchronization.
This further suggests that maybe these two domains should be merged and that this clock should really be part of the clk_i group. It’s something you’d want to investigate and look at more closely.
Techniques to synchronize between domains
We have various techniques to synchronize between designs like double-register synchronization, strategies for synchronizing buses which are more complex, and techniques for controlling reconvergence between signals. There’s also another technique which can be used if the surrounding circuitry has control of the receiving clock. If you can shut off the receiving clock, then you can guarantee that there’s no metastability if the data is only changing at a time when the receiving clock is shut off. So in some of those cases you can actually prevent metastability altogether.
Double-register synchronization
The simplest CDC synchronization is where you add a second register after the main receiving register. If the clocks are asynchronous, the first receiving register may in fact go metastable but most of the time it won’t stay in that metastable state for very long and hopefully it’s resolved by the time the second register clocks a value in and you get a normal value coming out.
This is ideal for synchronizing single bit values. While it is often shown with two separate registers, you can also just use consecutive bits of a shift register and synchronize it all within a single shift register.
Bus synchronization
Buses need better synchronization and the problem with double-register synchronization for buses is that if you happen to hit that metastability on the first receiving register in a bus, each bit changing resolves metastability separately, potentially leading to problems. Some of those bits may resolve to the new value, some to the old value, and the result for the bus is that you may get a mixture of old and new value bits, producing a value that is neither and may be completely invalid. One very common strategy for resolving this and synchronizing a bus is MUX synchronization where the receiving side says I’m not even going to look at this data until you tell me that it’s stable and safe. We use a separate, synchronized control signal here to tell us when a bus is safe to read. This is a very common way of synchronizing a bus to make sure that the values are internally consistent.
Another approach very common in FPGA design is to use one of the built-in FIFOs from the library of whatever technology you’re using. FIFOs are generally designed so that they’re prepared to synchronize between the write clock and the read clock on the FIFO and these are very helpful if you know that the data rates are differing on the two sides.
If you can prove that the input is gray coded and successive values differ by only one bit, then you know that standard double-register synchronization is safe to use for a bus. We also have some checks that use formal analysis to analyze the logic feeding into the input on the clock on one side to prove that at most only one bit changes in a given cycle. If the analysis comes back and says that two bits can change and here are the circumstances under which they change, you know you need to use a different synchronization method.
Controlling reconvergence
You may also have a situation where a single bit is synchronized into the receiving domain in more than one place. That can be fine but if they reconverge downstream, and this may be multiple cycles later, you have a problem. If you have metastability in both synchronizers, one may resolve to a zero while the other resolves to a one, yet they’re supposed to be the same value. So when they come together later in the logic cloud you have inconsistent values and you may get a result that you don’t expect because of metastability resolution. This again is something you can check for.
What about IP?
These days people don’t design everything on the chip all by themselves – they’re using blocks of Intellectual Property (IP) that come from other places like third parties. So you want to be able to do your CDC analysis of the part of the design you’re working on and know what the requirements are of the IP you’re using. On the inputs, what clock is the IP expecting that data to be originated from? Does it have internal synchronization that can say it doesn’t matter what clock it’s originating from? Likewise on the output side, what clock is that data originating from so you can check it against the surrounding circuitry?
This is something where we have some technology called User Grey Cell which annotates these black boxes with what’s essentially a model of the outer ring of logic up to the registers. It will tell the CDC analysis this data here is expecting this particular clock, this data over here is originating from this particular clock, and here are some feedthroughs that just pass the data straight through. We can model these cells even without knowing the full extent of the design which you may not know. If you do have all the detail of the IP, you can analyze it with the rest of your circuit but often the vendor may just provide a description or an outer shell of what’s available. The User Grey Cell allows you to do your CDC analysis and check your assumptions about what clocks signals are originating from and being received on.
Remember – fixing CDC violations changes the design
One important point to note here is that when we fix CDC violations by introducing synchronizers we are changing the design and this can affect the timing of your signals, the latency particularly, and the throughput. So you don’t want to wait until the last minute to fix your CDC problems. If you’re getting ready for tape out and you discover you need to introduce some synchronizers which can change the timing and the latency and so forth, that could invalidate a whole bunch of your prior simulations. So you want to start the CDC analysis at the RTL level if possible, to be able to fix and find your CDC violations early before you’ve invested a lot of time in simulation and synthesis.
Providing clarity to CDC
The Blue Pearl Visual Verification Suite is one tool that can help provide clarity to CDC, particularly for FPGA designers. It includes RTL-level CDC analysis early in the design cycle and an Advanced Clock Environment (ACE) which visualizes clocks and asynchronous CDCs in RTL designs to help users analyze designs for metastability. Along with its many linting and design rule checks, it lets you fix your CDC problems before you invest a lot of time in synthesis and simulation. Reducing debugging and respins, and helping you get to your design closure as quickly as possible.
This article is a summary of my presentation in a webinar hosted by Adam Taylor of Adiuvo Engineering & Training. If you’d like to know more, it’s worth your time watching the presentation – and listening to the fascinating Q&A session which followed and highlighted some of the CDC issues typically faced by developers and engineers. You can find the webinar at https://youtu.be/1L7HA5o3-2c.
