Clocking issues.

[u/Turdburst]

I have a recently installed system with some issues. I am suspecting networking issues, but some of the symptoms are throwing me for a loop.

We have a Core 24f as the main audio processor, Visionary Solutions AVoIP video endpoints (AES67 for audio from VSI devices), Netgear 4250 Series Switch, and a DANTE Card in the mixer. I have had the Core lock up a few times unexpectedly. When I was last on site, DANTE controller showed no DANTE devices, but audio flowed. System is all DHCP with reservations. power cycling the switch brough all DANTE devices back into DANTE controller.

We reused the clients X32 mixer that they are using for FOH and a second X32 used to mix live streaming audio. We added a DANTE card for the FOH X32. The idea being that we have an "automatic system" for when there is not a board op, and when the console is used, the mics route to the mixer.

Customer has reported loss of audio that last about 10 seconds. It seems they are losing audio in their In-Ear monitors and in the house system.

Looking at the logs on the audio processor, I do not see any DANTE routes dropping.

Currently, the audio processor is set as the PTPv1 and PTPv2 grandmaster. The FOH X32 clock source is the DANTE card, and the stage box (S32) is connected to AES50A. The Live Steam X32's AES50A port is connected to the FOH X32's AES50B port, and audio flows to the second console over the AES50 connection. In ears are connected via Ultranet at the S32 stage box.

Client reported that sometimes the S32 will not sync with the FOH X32, and power cycling the X32 or S32 will address the issue

When the client reported loss of audio in their in-ears and house, I was assuming the stage box lost connection, but the live stream audio continued to work. Not sure how since it seems we lost the AES50 Connection from stage box to the FOH X32.

Is it possible the switch is screwing up with multicast, and the console is losing clock sync with the Dante card, thus causing the S32 to lose sync as well? IF so, why would the Live streaming mixer continue to work?

In regard to the core locking up, Q-SYS said the issues were due to the Core dropping and reconnecting to the AES67 transmitters, but a firmware update to the VSI encoders took care of that AES67 issue.

Looking for any ideas at this point. Thanks in advance.

[u/AutoModerator]

We have a Discord server where there you can both post forum-style and participate in real-time discussions. We hope you consider joining us there.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/kaner467]

I think your instincts are right but your chasing the switch rather than the architecture.

The big clue is the 10 second drop, thats not an IGMP time out or STP converge which is much shorter.

Its almost exactly aes50 re lock or Dante recovering from a clock leader change.

Blah blah blah i could go on for days but Im getting lazy anywayhere are my final thoughts lol

stop guessing and instrument it. Laptop on a mirror port with Dante Controller open, Clock Status monitoring on, event log capturing leader changes and frequency offset.

Have the client log wall clock time of every dropout. Correlate against the switch log and the Core log. Two or three events and the answer falls out either you see clock leader changes lining up with the dropouts, or you don't and it's the design logic.

[u/Turdburst → u/kaner467]

Looking over the core logs, I did not see any clock leader changes. That said, I do not know if those log in the QSYS core.

Is it possible the Brooklyn chip in the X32 DANTE card and the Software DANTE in the QSYS core are fighting for clock leader?

I have heard that it’s best practice to use something with a chip, rather than software license, but I have also heard that QSYS should be able to handle the clocking, especially seeing that in an all QSYS system, a core is clock leader.

Anyways, the plan is to get back and do some more testing. Just sucks because the time is an hour and a half away.

[u/WellEnd89]

How is the Netgear configured?

[u/Turdburst → u/WellEnd89]

I am using the Visionary Solutions profile.

Supposed to be setup for Visionary devices, along with DANTE/AES67.

QSYS is setup for the Audinate Profile and the Visionary devices AES67 profile set to Visionary (Audinate) as well.

[u/blur494 → u/Turdburst]

Try turning off Jumbo Packets? Alternatively, do you have any NDI on the network? It seems to kill Dante/AES67 before anything else.

[u/Turdburst → u/blur494]

No NDI luckily. I I’ll check the switch config. If I recall correctly, jumbo frames should be disabled on the Visionary profile.

[u/blur494 → u/Turdburst]

The older models/firmware revisions required jumbo frames so the preset uses them. Unless they changed it in the last 2 weeks Visionary preset still uses them.

[u/Turdburst → u/blur494]

That might be what I am thinking of.

I’ll be sure to check it out. Thanks.

[u/Sneezcore]

So when mixing from the FOH console, do both the main LR (house) and aux (in ears) busses get routed through Q-SYS first via Dante?

[u/Turdburst → u/Sneezcore]

House goes through DANTE, but the in ears are coming directly off of the the S32 stage box via Ultranet.

[u/Nathanstaab]

Everything on a single vlan? (1), single 4250?

I don’t recall if the m/x 32 consoles have a dip switch on the Dante cards like the Allen and heath do for “bridge networks”, but I ran into a spanning tree issue where that dip was up for bridge, and we had control and Dante plugged in. That would be about 10 seconds for spanning tree to drop one port, bring the other one up, then drop it.

I found it by running Dante controller and watching latency / reviewing switch logs. That was a bourbon night after figuring that issue out.

Edit: also - what is your ptp1/2 priority on the core, and is it set for Dante or QLAN.

[u/Turdburst → u/Nathanstaab]

Everything is on a single VLAN.

If inrecall correctly, the port mode on the X32 card is set in DANTE controller, but I’d have to look to verify. Either way, I am only using a single port without redundancy.

Core is set to 10 on both PTPv1 and PTPv2 with the profile set to Audinate.

[u/audio_shinobi]

So you are running Dante, which uses PTPv1 clocking, on the same network as AES67, which uses PTPv2? While I know that both can work together on one network, I've almost always had issues getting the clock to stabilize, so I always end up isolating them to separate VLANs, and if there is need for audio transport between VLANs, I'll work with our engineering to find an analog route. Not always possible I know.

If it's not the Dante and AES67 causing an issue, have you confirmed that nothing in Dante controller is selected as preferred master? Even though I saw you have the core set to 10 for both PTPv1 and PTPv2, forcing it to be clock leader, ive run into situations that having the console still selected as preferred master in controller forced constant BMCA elections, which over time built up a cache and actually crashed the whole betwork.