Crazy new problem has surfaced. 1year cert renewal process doesn't work anymore
Just last night everything in my Control4 system stopped working. The remotes are stuck on, "waiting for network". The control4 app on my phone won't connect to the controller and tells me I need a 4sight subscription. None of my other macros and procedures trigger anymore.
I've had this system for several years and I've been managing it on my own for quite some time. This blog has been a life saver from the day it was setup.
I'm currently running on 2.10.6. I can't upgrade past this point due to the fact I'm using HC-250s everywhere. I have three of them total.
I figured the problem was that it's time to do the 1 Year cert renewal. But, I did the 1 year cert renewal only a few months ago in November 2021. Well, I did it anyway. I followed the Definitive guide step by step. In fact, I've gone through it three times now.
I uninstalled all versions of composer and cleaned out the respective folders to start fresh.
I can confirm that I get all the necessary certs generated under /users/myname/appdata/roaming/control4/composer when I run the special composer.exe. I then SCP the clientca-prod.pem into the /etc/openvpn directory on the controller and then reboot it. I also concatenated the clientca-prod.pem as described in the instructions.
From there I attempt to access the controller using 2.5.3 and I get an error message saying, "unable to connect to a director at xx.xx.xx.xx on the unsecure port". Which, I think makes sense since I do believe I had set it up to use the secure port (I don't remember what the actual setting was called).
So then I install composer 2.10.6 and attempt to connect again and I get the same message. Only it doesn't say anything about the unsecure port.
One peculiar observation is that when I open System Manager, none of the controllers (master controller or otherwise) auto-populate in the list. I have to manually add the controller by clicking the add button and specifying the IP address of the controller. I've always found that if it doesn't auto-populate, or discover the controller then there's something genuinely wrong. I disabled my firewall on my machine just to see if whatever discovery mechanism it uses is being blocked but that didn't help.
Once connected to the master controller in System Manager, everything appears normal. The expected services are running, including director and broker.
I've also logged into my control4 customer account to verify that the controller is still registered. It's most recent check-in was last night at around 1:00 am.
At this point, I'm not sure what to do. I do have a recent backup of my project, so I suppose I could factory reset the controller and truly start fresh but I'd want to leave that as a last ditch effort.
Any suggestions you have would be great!
Try reboot of network and all controllers
brians2803,
I really appreciate you responding to my post. Any support I get is awesome. I'm wondering if you could maybe provide a little more of what I should be looking for when I've rebooted the controllers.
With regard to rebooting the network. I think I need to provide to clarification to my setup. I have a dedicated router, dedicated wireless access points, several cisco switches and a dedicated internet modem. My network is further subdivided into separate networks (broadcast domains). Routing between networks is obviously facilitated by my router; a pfsense virtual machine.
However, with that said, all of my Control4 equipment, and the virtual machine that I'm using to run composer, are on the same network, within the same broadcast domain. So all devices within Control4 should be communicating at layer 2. So in terms of rebooting the network, are you suggesting I only reboot the switches providing layer 2 within that broadcast domain, or are you asking me to reboot the entire system in my house?
As an alternative, I am comfortable with Wireshark and tcpdump, is there a specific type of network traffic flow that I could look for and confirm is actually flowing between to/from the master controller and my machine running composer? Is there a specific network socket that I can confirm is being opened and established between the devices? Or perhaps a specific service and its logs that I can tail to identify errors that are occurring?
Given the error message I'm receiving, "unable to connect to a director at xx.xx.xx.xx", would there be a way specifically troubleshoot the communication process that occurs when establishing a session with director on the master controller from composer? My suspicion is that it's still an SSL (certs) type of problem, but I don't know how to troubleshoot those kinds of issues.
I'm going to go ahead and reboot all of my controllers. That's a great idea. I've only been rebooting the master controller up until this point. I'll report back when that's done. Thanks!
Some interesting new details. But before I share them, I do want to mention that I did reboot all of my controllers but observed no improvement.
In my previous response I had mentioned troubleshooting the handshake between composer and director. So I started doing that. I installed Wireshark and started capturing traffic to/from my master controller starting at the precise moment when I click the connect button in composer. Attached is the picture of what it captured.
According to what I can see, composer sends the initial tcp syn packet to the master controller on the destination port of 5021. The controller immediately responds with a RST/ACK. Meaning it's declining to establish a session.
At this point I SSH'd into the master controller and checked what LISTENING ports are open on the controller using netstat -plnt. I've attached a picture of that as well. It doesn't look like there are any processes listening on that port. In fact, it doesn't list director at all. Though, I recognize that it's possible that some other process could act as a proxy and forward requests to the director process on a different port. But the point is, NOTHING is listening on that inbound port number on my controller. Which would explain why the controller would simply ignore any tcp syn packet it receives to that destination port number.
I then went into System Manager and confirmed that director was running. Attached image.
I did the ol' stop/start of the process anyway. It looks like the process started with pid 16678 but then after a few minutes it stopped. Director is no longer found the the process list.
Anyway, there's some new information. I haven't solved anything yet, but here's some interesting details I'm discovering.
Uploaded files:I have had problems when the sub net is any thing other then 255.255.255.0 if 255.255.0.0 it won't communicate with controllers check the sub net
Just to close out this post, I just wanted to state that in the end I just factory restored my master controller and then restored from backup. Despite having a recent backup it still took several hours to get back up and running. I had to rebuild my zigbee mesh and then re-join all of my zigbee devices. I also had to re-add all of my alexa triggers. What a pain!
After it was back up and running I confirmed that, in a working system, the master controller does indeed have a director process listening on inbound port 5021 (plus two other ports). So there was definitely something hosed with my master controller and the fact that there was no open socket for director to listen on. I suspect I might have been able to save myself some trouble if I'd looked closer at the director logs. Something would have indicated an obvious error with director somewhere.
I've had something similar happen to me on a HC-250. Factory reset fixed it, but then eventually (months later) it crashed and was unsalvageable. I believe my issue was likely ram or SSD that had gone bad and corrupted critical files. A factory reset fixes those files, but if the hardware is bad it will eventually corrupt things to the point that a factory reset won't work.