After Covid Coma

[Bob]

The first op session since the lockdowns was held on Oct 16, 2021. It was a slow session that was plagued by a lot of gremlins since the O Scale National in Denver and the first real op session in almost 2 years. Still, we had a lot of fun. I only took a few quick phone photos since I spent most of the day troubleshooting and repairing some of the stuff that had mysteriously broken.

It was Levi's first time dispatching with a US&S hardware CTC panel and a train sheet. Due to the gremlins he had a slow day that made for low-stress learning. The CTC system acted up due to some sort of failure with one of the SMINI cards under the layout requiring Levi to talk trains past two sets of false occupancy detectors that caused many red signals.



Jeff brought two of his beautiful Sunset SD9s in Burlington livery.



Rick served as the 16th Street yardmaster. Here he explains operation of control panels at the south end of Havens Yard to Roy.



There was an oopsie as a car picked a switch point in Ricksburg Yard resulting in a coal spill blocking the passing siding.



Darwin was an experienced operator, but not on the A&O. He had his arm twisted to run the yard in Linnwood. Although he was told it would be a "quiet switch job" he was slammed with 4 trains almost at once, each requiring attention in the yard. We'll see if he ever comes back... Linnwood is also known as the "rain room" in which there is a perpetual thunderstorm and the sun never shines. Attend to personal needs before starting this job...



Before the National Convention, David erected a lot of red rosin paper hardshell forms for scenery. Here C425 #475 returns with the Scarlet Shifter from the Kayford branch with empty hoppers.

[Bob]

One more from the session. LeRoy switches the 34 linear feet of International Paper with SW1200 #88.

[Craig]

Wait…is Dispatch reading a magazine too?  . Hehehhee

Still looks like everyone had a great time. Thanks for the shots Bob!!

[Bob]

Yes, Craig. The dispatcher is reading a magazine. That's actually closer to prototypical than most A&O sessions. Many real-life dispatchers refuse to dispatch model layouts because "There is no time to think." When the dispatcher gets behind on the A&O train sheet, he will die. Don't ask how I know.

It was a slow session. The last op was about 20 months ago. A lot of regulars couldn't make it, and as for the substitutes and the rest of us (myself included), loosely paraphrasing a famous line, "Forgive us, for we know not what we do."

To top it off, one of the C/MRI boards decided to partially fail and give false occupancy readings that the dispatcher had to talk around. Two key tortoise motors in Ricksburg came down with problems, one a short, one inoperative. Both have been repaired by David.

One NCE Cab04 died and another received a sheared off antenna (I didn't judge who dropped it. I just calmly repaired it during  the session.)

We ran out of throttles and that was a problem, too. Not that we didn't have enough, but that the NCE system was overloaded trying to serve all those throttles. The polling rate, indicated by the flashing red LED on the top of a radio throttle, was between 2/3 and 1 second. I plan to count how many we have then confiscate 1/3 of them for the next session. No throttle? Wait your turn. And turn in your throttle when you complete your assignment.

Everyone except David and myself still seemed to be enjoying themselves. We were bummed by all the 2-year downtime gremlins. Nothing works better to keep problems at bay than frequent op sessions.

Earlier this week I spent 7 hours working on the CTC failure. The SMINI in Linnwood has been replaced with a spare and a new brain transplant processor will be installed prior to Turkey ops. It should be operating fine by then.

By the way, I'm not sure why there are two of the same photos in my last reply. The second one seems to be an inline attachment, as clicking on it doesn't open the usual PostImage window. Hmm... I suspect we shouldn't be adding inline photos since a separate hosting service has been set up for them to keep the main A&O forum disk footprint small.

[Craig]

Talked with a few of the guys…they all said it was great.

Any thoughts on the SMINI issues Bob?  I’ve been thinking about the issues, but the standard thoughts that come into my brain are voltage fluctuations, grounding / noise, or bad chip. I know what we did on those boards…so the chip thing comes to mind.  You have any thoughts?

[Bob]

The CTC failure was a dumbfounding mystery. Still is. I was thankful that it wasn't an intermittent failure, but more on that...

The SMINI has 3 74LS540 octal buffers to drive the 8 bit bus to read 24 input pins. After voltage readings proved that the detectors were fine, I replaced one of those chips assuming that somehow it failed. No joy.

Both false detections were curiously of the same bit on the processor bus. But the second of 3 input buffers read correctly. All should were high on the detector inputs and low on the inverting buffer outputs (again, they were.) I verified this with my 100 MHz Rigol DSO, triggering on the read pulse to each buffer and measuring the input pin at the processor. Everything was perfect. Not noisy, not close to a voltage margin. The code, after asserting the read pulse to a buffer, waits a full microsecond for the bus to settle before reading the bits. That's way overkill, but causes no issues.

What I could prove over and over again was that the processor board bus pin saw a logic low (no detect) but the CTC machine was told high, (detect.) So was it code? Nothing changed over the months of flawless operation, through the O Scale Convention, up to the latest op session.

Late in the evening I once again hung an oscilloscope probe on the processor bus pin and suddenly I heard the CTC machine receive an indication code. Say what? The failure disappeared! So it had to be something with the SMINI.

At that moment I swapped out the SMINI and I will install a new brain transplant processor the next time I'm at the mothership.

It was working properly when I left. Film at 11:00.

[Bob]

In case anyone is curious about the SMINI brain transplant, I made an adapter board so that I could program a PJRC Teensy 3.5 processor to replace Chubb's original PIC processor. That let me add more robust error detection. We've upped the RS485 bus speed from 28,800 baud to 115K baud with robust operation. Until this fiasco.

The Teensy 3.5, an Arm Cortex M4 processor, and adapter board:


The first installed brain transplant at Glenn Forge, running at 115K baud. There is a SparkFun Teensyview OLED display piggybacked on the processor for diagnostic information:


Reminder: click on a photo to enlarge the thumbnail. Click again to see full resolution.

[david]

I have absolutely NO idea what you're all talking about!! What a lucky guy am I

Katie says I need to treat you all very well. I do my best.

The "Grand" Poobah

[RickBacon]

I'm amazed at the complexity of the A&O signal and control systems.  Even more amazing is the reliability of it all!  Thanks Bob and Craig!  It is my experience that a well designed and built, complex machine can seem to be alive.  Just like a human, when the machine is idle, it begins to decay!  I think this op session was a realistic shake down run for the machine coming out of hibernation and back to life!   

David, I'm sure the rest of the crew would have to agree with me, we owe you equal appreciation for allowing us to be a part of your dream! 

[Bob]

Rick -

Don't omit Vince's contributions. He generously provided the assembled SMINI boards and Chubb occupancy detectors for the CTC system, and he assembled some of the banks of detectors and block cutout switches. He also installed the first fully-working signal, an automatic dwarf in South Ricksburg.

The more I ponder things, I'm leaning towards blaming the replacement processor I installed. All the signals on the SMINI looked perfect when probed with the oscilloscope. It is unlikely to be the programming, since it had been working perfectly as the other 4 continue to do so. And, the oscilloscope showed the Teensy 3.5 processor received a no detect at the input pin of the PCB but it reporting detect. The 3.5's processor is in a BGA package (ball grid array). Solder joints can crack from flexing the circuit board. A replacement is sitting on my desk ready to be plugged in. Assuming the problem doesn't come back, I'll just toss the 3.5. I have more.

Another failure that day was the 408 decoder in the high hood N&W GP35 #213. It is working now on my home NCE test track, but I had to do a decoder reset twice and then reprogram it. My spidey sense tells me that it will probably conk out again. If so, that'll be a good time to do a full monty and install a Loksound L with light package and mechanism C/L/A.

A project planned for this fall is add a radio-linked mini CTC machine to control signals at the New River diamond. Why radio? There are not enough free I/O pins on the Morrison SMINI.

[Craig]

Hey Bob.

The BGA is actually a good thought. Been experiencing strange, downright weird behavior on a new “bus” comm protocol. Anywho..just like you mentioned, we use a BGA on the assembly. We started to do some troubleshooting and found that we were getting watchdog faults. Look at the lines specifically and it looks good. So we start to apply some thermal cycling and boom!  Failure.

Upon inspection (both PCBA build and unit installation) we believe that the board is getting flexed when connecting to our high density bus connection. That gives enough friction when installing, when the board is handled on the outside it could cause flexure failures. We changed the installation instructions and have see a huge drop in issues.

[RickBacon]

Bad on me! I totally forgot about Vince! He has most definitely been a big contributor to the A&O electronic department! If I remember correctly, he did most of the signaling work on A&O 1.0 also!

[Bob]

Craig -

What's really odd is that the signal levels were perfect at the male header soldered to the processor board. All three octal input buffers on the SMINI were at +5 on the inputs and pulled down to a perfect logic low when read. But the first and third failed, reading high inside the processor as indicated by the data stream sent back to the CTC machine. That doesn't sound to me like a cracked open connect on a BGA pin. And that same pin drove 6 output CMOS buffers just fine, though I foolishly didn't scope them to see what logic levels it was driving when switched to output. Perhaps a short to something else? Ever heard of that happening under a BGA? The processor is only a couple years old so tin whiskers from RoHS solder would seem unlikely.

My only other experience with a bad BGA was on a network board inside a donated Laserjet printer. That board, one of the first made in the Philippines for HP, had a notorious failure rate. I tried reflowing the part with a heat gun. The repair worked for about a week, then failed again. I gave up. I could still use that printer through a USB port to print home-etch PCB patterns on transparency film, but my second heated etch tank cracked and I'm done with home etching. Time to learn KiCad?

[Craig]

Hey Bob.

Shorts to something else?  Maybe. I suppose…if the BGA could have connections close enough to other pins. Like ground on a pin nearby. But doubtful on this. At least the experiences I’ve seen with BGAs, it hasn’t been in lateral breakdowns.

Now on multiple layers..that could be something. Maybe we are seeing breakdown between signal layers?  I’d assume that these are multi-layer boards?  On the ones we had at work it was micro-racking between the layers. It really showed up in thermal conditions. Do you think we could be seeing things like that?

[david]

Hmmm. Possibly. Anyway you look at it we're taking it in the shorts.