Communicating Between Rockwell Controllers Part III: Redundancy

Part 1 and Part 2 discussed UDT optimization and simple communication between Logix5000 (CLx) processors. This last part concludes with CLx↔CLx communication methods for redundant network channels and/or processor pairs running enhanced redundancy. Redundant channels can be used to increase equipment availability despite down equipment or to unload primary processor communication load.

Redundant Channels Between Two CLxs

Some installations have redundant channels between CLxs. Rockwell handles DLR (device level ring) redundancy inherently, so no special programming is necessary. But for systems with multiple Ethernet ports attached through separate networks, considerations must be made for how to configure the redundant communication. All of the options listed below include a separate MSG block and data age timer for each channel.

To be useful, redundant systems must warn if they are partially down so that the down channel can be brought back into service before the last remaining channel goes down. Options include:

  • The two MSG blocks in each processor read the same tag in the other processor to the same local destination tag.
    • If either MSG is successful, new data will appear in the destination tag, updating its heartbeat and RESetting the common data age timer. The DN (done) bit from each MSG block detects that down channel: when DN comes on, unlatch (OTU) it, and RESet the channel data age timer for that channel.
    • This is simple, effective, requires minimal memory & programming, and is adequate for many applications. Note that three timers really are necessary - if the other processor is communicating, but not running (in program or faulted), both channels may report DN and reset their data ages, but the HB will not reset the master data age. That is still cause for alarm and equipment trips. The channel-DN data ages are for loss of redundant backup warnings only - trips should come from the HB data age.
  • For applications where it is too critical to detect a lost channel to trust the DN bit approach, two of many additional options are:
    • The source processor has two copies of its tag. It continually copies all the data from one to the other, but has distinctive HBs for the two tags (even vs odd, 1-50 vs 51-99, etc). The MSG read one from each source tag to the same destination tag, and RESet the data age for each channel when the HB changes to a value matching the one for that channel. For this to work well, the two MSG blocks must be triggered at different times.
    • The source processor has only one copy, but the destination processor has two copies. The MSG reads to each destination copy. When the HB on the second copy changes, the processor copies all values other than the HB from 2nd copy to the first. This approach does not require synchronization between the MSG blocks. If the UDT is not arranged as I recommend in parts 1-2 with a SINT after the initial HB, 3 local copies will be needed - one for each channel, each copied whole with the HB to the working copy when it updates.

When handling redundant communications with any of the above methods, it is critical to keep each tag ≤488 bytes so that the entire tag is updated at the same time. Otherwise, the HB might update when some of the data has not updated. If >488 bytes needs to be transferred, set up as many separate UDTs and pairs of MSG blocks as is required to keep each message ≤488 bytes.

Reading Data from a Synced Redundant CLx

To unload connections from a heavily loaded CLx in a redundant pair of processors: read from the synchronized backup (“BU”) instead of the primary. Its IP address will be the same as the primary, with one added to the last octet (ex. 192.168.0.11 would be the BU for 192.168.0.10). Downsides are that the system will still occasionally revert to communicating with the primary, and updates from the BU may be less frequent, so this is not advisable for time-critical data. The delay depends on the size and nature of the project - test to find out what it is in your situation.

The CLx reading from the BU will need to be configured to also read from the primary if the BU goes down or has to re-qualify. If the data age from the BU’s HB rises above some minimal time - I recommend <⅓ of the warning timeout and certainly <⅓ of any trip timeouts - the MSG block reading from the primary would be enabled. The MSG block from the backup should be enabled all the time. However, while I normally ignore DN and ER bits, in this situation because there is a BU communication channel (to the primary) always available, the trigger to the BU should wait until the MSG raises either DN or ER.

Obviously, there will need to be two MSG blocks and two instances of the UDT, one for the primary and one for the BU. COPy data from the BU instance over the primary when its HB changes. Unlike the situations described above for redundant channels - in which HBs need to be kept separate - in this case you want the HB in the main instance to update when either MSG read is successful.

Read from BU | Cross Company

Summary

Take advantage of redundant network channels and processors with multiple read MSG blocks. Lost channel warnings are critical to high-availability redundancy. Primary CLx connections may be unloaded by reading data from synchronized backups.

Contact Cross Company

Chris Hardy is an electrical engineer out of Georgia Tech. At Cross Company Integrated Systems since 1994, Chris has process control experience with boilers, alternative energy, water/waste water, chemicals, pharmaceuticals, security, textiles and automotive. Chris also programs controllers/HMIs and writes custom windows applications for communication, data collection, display, trending and reporting.

Topics