Elmer’s Weekend
Hi, my name’s Elmer, and as part of the PROCENTEC Support Team, it’s our job to support a network of engineers and network architects across the globe, helping keep their systems fully operational 24/7. Every day we come across a variety of problems. Some are easily solved, but occasionally we are inevitably presented with a problem that can cause us to scratch our heads, and force us to think out of the box — A recent case springs to mind.
A Downtime Nightmare
Located 3 hours offshore, in the middle of the North Sea, is a vast array of wind-turbines that, at the heart of their systems, has a PROFIBUS network essential to keeping the flow of electricity to their batteries.
On this occasion, we received a rather urgent call from their engineer. The network had failed, no amount of in-house expertise could diagnose the problem – could we please help? It was 16.00 on a Friday, our general office staff and engineers were starting to wind-down and say their farewells for the weekend. But of course, our support team remained on-hand and well prepared for the fact that network failures don’t tend to occur on a 9-5 schedule.
We ran through a fast diagnostic checklist with the wind-turbine engineer, probed their network remotely, but were also unable to pinpoint the fault from our Netherlands location. Could we come to them, they asked? 3-hours offshore. No problem.
I said, “I’ll check for hotels and flights now, and jump on a plane Monday morning.”
They said, “No, We will send a car now, and have a helicopter waiting for you at Groningen tonight…”.
It was an understandable response. With the electrical feed down, not only was the client losing revenue but also incurring enormous penalties for failing to fulfill their national grid commitments.
Troubleshooting Onsite
By midnight, I was shaking hands with the nervous group of engineers, standing on a very windy and rainy platform in the middle of the North Sea. Time to show them what we do best – Diagnostics.
After re-confirming the obvious questions. Have you changed anything? No. Have you had any other power problems? No. I set about connecting my PROCENTEC ProfiTrace diagnostic tools to the network. Seeing multiple illegals, I suspected an EMC problem, but upon measuring all segments, to my surprise, it wasn’t. Then, I consider a device was the problem. But again, the illegals were not related to any specific device.
Digging deeper, I studied the drawing and could see an odd configuration where two cables led back to a single connection but unapparent why. The engineers took me to the location, and upon inspection, I could see that what was supposed to be the last device, was not last. These two cables connecting to each other had created a copper ring.
The Solution
While it is possible to have a ring on a PROFIBUS network, I could see no component, such as an OLM that could handle this type of functionality. Confused, I asked again, “Has it always been this way? You haven’t changed anything?”.
“Ah, yes…” came to the reply, “We had a termination there, but it was damaged, so we managed to connect that cable in the free port of this connector. But that was weeks ago.”
Reading this, it may seem obvious now, but errors like this are commonplace and often have no immediate effect on a network. Sometimes the mistake remains unnoticed, until a secondary system or component becomes online, resulting in a network outage, and for no apparent reason.
After putting the termination back, the network was successfully restored, and I was whisked back home to enjoy the rest of my weekend. Monday morning, in the carpark, I bumped into our CEO, Pieter Barendrecht. “Hi Elmer, had a restful weekend?” he asked. “Well, since you ask…”
Tech-tips by Elmer Vis & Chris Simpson
Procentec Support, Training and Testlab – Netherlands