+12
Started

Portal not responding

Phil K 3 months ago updated by Craig Silver 2 days ago 46
ConnectWise Control Version:
19.6
Server Affected:
Host Client Affected:
Guest Client Affected:

Answer

PINNED

If anyone with issues is willing to help us debug, you can run the server in debug mode like so:


sudo /etc/init.d/screenconnect debug > debug.log


Then send me your system info (CPU, RAM, VM platform if applicable, reverse proxy setup, distro), your web.config, and the debug.log file: edavis@connectwise.com

Usage information might also help (how many sessions of what type, how many concurrent sessions you typically have, any correlations you have observed between your issue(s) and server usage, etc.).

The Mono server is definitely in a rough spot right now. On top of the bugs we've found in the Mono updates with 19.5/6, some of the changes we've since made in the server (esp. those aimed at improving performance/scalability) don't seem to get along with our Mono fork.

Some of these issues also don't show up in our internal testing, so we're working on a load simulator to help discover any issues that only become apparent with higher usage. (This would have helped us find that big memory leak in 19.6 a lot earlier, for example.)

after upgrading to the latest Unix version, the portal stops responding after a couple hours. I have to reboot my server for it to start working agsin. 

This is no small bug but, after two weeks, its status is still "Under review". Please clarify what "Under review" means.

Started

It means I forgot to update this. (We've been working on Linux-related issues since they were reported. This one looks to be a memory leak.)

I was told by the chat support that my issue is related to this one. As I am logged into the host portion the website continues to work. If I have an on the fly support session created, the user only gets a blank white website when going to my main URL. Rebooting works. I've experienced this off and on at random as of a few screenconnect versions ago.

Hi Eric, how's this one coming along?

Is the 19.6.26659 release supposed to fix this?  It's unclear to me if the release note line item "Bug - Mono Server - WebClient throws attempting to make secure requests" is referencing this issue or something else.

It is not - the fixed bug is for outbound web requests requiring TLS connections made by the Control server failing to complete (e.g. outbound SMTP requests, the status page checks, etc.)

The one on this page is still being worked on by development.

As a developer, myself, I understand. Eagerly awaiting a fix! Thank you.

I'm not sure what is taking so long with this, or why the Linux releases keep getting pulled.  Now the latest Linux release that can be downloaded is v19.4 from early November.  I get that the primary focus is Windows, but there are those of us that depend on a Linux on-premise build and don't want the complications of having to add a Windows machine just to handle a single app.  With Microsoft's push to make .NET more and more cross-platform I'd think it would become easier to support Linux.  Maybe it would be worth looking into .NET Core to unify the codebase and replace Mono entirely?

v19.4 was the last really stable version for me,

Asking again since another Linux-related bugfix has been posted for v20.1.  Is the bugfix for "Memory leak in Mono server" related to this issue?  It would be nice if the output stream could reference bug reports so we would know if there is any connection.

Unfortunately, we've heard reports from Linux admins trying 20.1 that something else is also causing hangups, even with the fix for the memory leak.

@davison that would be great to know. Can support respond? 

Is there an ETA on the fix? Can the fix from 20.1 be backported to 19.6 since it sounds like 20.1 introduced other issues? Is there some logging we can enable to help move this forward? We use this (or try to) every day and after this last upgrade, it's almost unusable. If we restart the server as Phil K mentioned above, it still takes 15-30 minutes for the web interface to start responding.

We updated to 19.6.27027.7360 on the 26th. It still has the same problems and goes down several times a day, but it does recover faster.

Yeah, unfortunately i think the other hang up issues Eric mentioned people encountering on 20.1 were already present in 19.6, its just they were overshadowed by the memory leak. On the plus side it should mean that 20.1 at least hasn't introduced any new regressions in that regard.

I'm losing my hope on this. Here's what I got from chat support: 

Routed Queue : Support - ConnectWise Control >> Support
When will ScreenConnect_20.2.27450.7387 be release for Linux because v20.1.27036 is not reliable
02:16 PM - lslater: Hi Gilvan
02:18 PM - xxxx@gmail.com: I"m running ScreenConnect_20.1.27036.7360 on Ubuntu 18.04.4 LTS and having this message "Internal Server Error" when I try to log in several times a day. I have to reboot the Server and sometimes I can login but the problem comes back later on
02:20 PM - xxxx@gmail.com: I was hoping that the Stable ScreenConnect_20.2.27450.7387 would solve the problem but there is no Linux version only windows. It looks like Linux support is not important anymore. Is that the case from now on?
02:24 PM - lslater: My apologies for the inconvenience. I have no had a instance of this happening unfortunately so it does not seem to be a common thing.

I see that ScreenConnect_20.1.27036.7360_Release.tar.gz is now "stable". Does it include a fix for the 20.1 hangups that were reported earlier or should we wait for 20.2 to stabilize? Please advise.

I tried 20.2 for two days, it was fast, didn't need any web.conf modification to work with nginx proxy /just had to comment out the host line/ but after some time it slowed down, for some reason the mono processes started to consume the cpu. The symptoms were the same as with the memory leak but htop showed a different kind of problem.

Hi there. 

On premise Control here in a Debian Stretch, old stable, VM. 

I tried several time the upgrade, but everytime had to revert to 19.4, which is the last stable release for me. 

I start to feel frustrated. Also, I'd like to know how many of us are using on premise connectwise control with Mono/Linux, because I think that i's the only way to go on cloud premise.

I'm starting to think Connectwise wants us to switch to their cloud service. 

Just my thought, anyone feels the same?  

 

The only supported distro now is ubuntu.  

debian jessie here and having exactly the same experience. in the last let say six month I upgraded my install at least 10 times with different version just revert back to the working 19.4

I upgraded to 20.1.27036.7360, also running on prem linux. Seems to be worse than the previous release, if that's even possible.

To add to this, when can we actually get built in ssl support??? It is 2020 after all

the 20.x series has upgraded/current ssl support.  I run it direct right now without a reverse proxy.

Any documentation somewhere on how to get SSL setup, step by step?

This is absolutely unacceptable that the product has been unusable since mid-December. I've been a huge fan since the ScreenConnect days, but will no longer recommend this product. My company will almost certainly switch to a more reliable product when I leave my post next month. 

Personally I don't believe I'm having any of these issues (or the crazy CPU utilization) with my 19.6 installation on a Ubuntu 16.04 x64 machine (currently about 130 access clients online).  I am running a NGINX reverse proxy in front of the CC instance--not sure if that explains why I'm not having issues... It just makes me scared to mess with my install by upgrading or changing config with all the reports of bad stuff happening.  I just wish there was more transparency about all the data.  I'd be willing to help with some diagnostics if CW would give us some debugging steps to run so we could get more data points.

@Davison If you're not having any issues i definitely wouldn't change anything. Our setup sounds the same as yours with Control 19.6 running on 16.04 LTS x64, behind Nginx, and we have been getting non-responsiveness multiple times a day since we upgraded from 19.4 a month or two back. 

PINNED

If anyone with issues is willing to help us debug, you can run the server in debug mode like so:


sudo /etc/init.d/screenconnect debug > debug.log


Then send me your system info (CPU, RAM, VM platform if applicable, reverse proxy setup, distro), your web.config, and the debug.log file: edavis@connectwise.com

Usage information might also help (how many sessions of what type, how many concurrent sessions you typically have, any correlations you have observed between your issue(s) and server usage, etc.).

The Mono server is definitely in a rough spot right now. On top of the bugs we've found in the Mono updates with 19.5/6, some of the changes we've since made in the server (esp. those aimed at improving performance/scalability) don't seem to get along with our Mono fork.

Some of these issues also don't show up in our internal testing, so we're working on a load simulator to help discover any issues that only become apparent with higher usage. (This would have helped us find that big memory leak in 19.6 a lot earlier, for example.)

Hi Eric, I sent you an email with debug logs few weeks back and haven't heard anything. Just wanted to check if you received it?

Hi Eric, I don't know how I didn't see your request for debug info. If it still helps, I will send mine. Let me know, and let me know which version. I might have missed an important differentiation between 19 and 20--not sure why they are both listed as the "stable" versions. In any event, let me know if I can still help. FYI, I am currently still on 
19.6.26378.7317.

How long would you want the service to run before sending the debug log? Even though my 19.6 instance seems to be working OK I thought I'd go ahead and send the debug info in case it helps with the diagnostics.

Why did you fork mono?  why not use mono that's out there already?

+1

@wwarren I'm sure they had to fork it because the standard version isn't totally compatible with their codebase and they needed to add some fixes.  That being said, it does introduce a large maintenance burden which is why it took this long to get any Mono upgrades at all.  I still think the best long term solution would be to migrate the entire codebase to .NET Core since Microsoft has made a commitment to cross-platform compatibility and things are moving even more in that direction with the next version of .NET (v5).

This is out of control. The tool I use to support my customers stops responding every 10 minutes

I noticed today that ScreenConnect_20.2.27450.7387 is now stable but for Windows only. I've been waiting anxiously for this stable version for Linux since the so-called stable version 20.1.27036.7360 has been crashing several times a day.

Are you guys abandoning Linux support moving forward?

+1

Is there any work being done on this or internal progress being made? We're really stuck in a bad spot. I sent logs and server info to Eric but didn't hear anything back. Are we being abandoned? It feels like it. What are other people using in the meantime while this is not working?

Eagerly awaiting a fix for this, too. Any update would be much appreciated!

We're working on it. We're having to work around some performance enhancements we made (again due to Mono incompatibilities), so that has introduced additional delay.

Do you still need more debug logs ?

Same question: Could you use more logs? FYI, I'm on 19.6.

I think we have enough for the moment. (At least on older versions.)

Once we've addressed the known stuff, I'd appreciate it if you guys would give 20.x (not sure which version is going to have the fixes yet) a go and send in the same info if you still discover issues.

(We should also have better internal testing in place before then, but of course we can't test configurations we don't know about.)

Is there a way to centrally turn off auto-update of all hosts so that, if we must downgrade the server after testing the latest and greatest, then hosts will still be compatible with it?

Yeah, if you're using the Advanced Configuration Editor extension:

Admin/ Configuration / Web Configuration / Quick Settings / Automatically Update Agent Version -> uncheck

Perfect! When ready, hit me with whatever you got.