A public pre-release of 20.2 Linux is available for download here. Please check out our Linux specific output stream for details on that release. We also have a separate page for Linux bug reports. This thread will now be closed.
I haven't posted an update because things remain largely the same as when I last posted. The issues you all care so fervently about are still going through review and QA. I'm checking now on the status of a possible test or canary build of 20.2, and will post here when I know more information.
We’re nearing completion of our internal QA, and should have a test build ready soon. To get access to this beta build, you’ll need to sign up here. Please note that this is closed testing, available only to those that sign up.
Have a great week,
As requested, I'll keep Mono-related updates in this post. Updates are posted on Mondays for the previous week.
* July 6: Fix submitted for cert chain issue; preliminary testing of new 20.2 build; verifying compatibility with newest Ubuntu LTS release
* June 29: Continued working to resolve cert chain issue
* June 22: Found and fixed web server restart issue in the course of investigating DB maintenance bug; working on certificate chain issue
* June 15: Continued on DB maintenance bug
* June 8: Wrapping up some other work, so engineering should have more bandwidth; fixed extension loader bug; started work on bug with editing DB maintenance plans
* June 1: Engineering was mostly diverted to general server bug fixes and other tasks; should be able to get back to Mono issues this week
* May 25: Working on general server bug fixes (i.e. not Mono-specific); large rewrite of service life cycle-related code that should address some long-running service restart issues (and hopefully works well running on Mono)
* May 18: Fixed post-setup hang/crash bug; working on extension loader bug
* May 11: Fixing 20.2 for the next release; language version-related issues appear resolved, fixing other known bugs now
* May 4: continuing runtime/compiler update; due to further Mono-breaking application changes in 20.4 (and the possibility of more to come), decided to try and fix issues in the 20.2 release until the breaking updates are complete
* Apr 27: updating Mono fork to resolve issues with newer C# features used in the application; work temporarily suspended mid-week to resolve escaped bugs in 20.3 release (I believe these are all resolved and in testing now); floated suggestion about back-porting fixes (no promises yet)
We appreciate your patience as we continue to work through these issues. We had some changes to our team, but I'll be providing updates moving forward.
I can see how you all would think there’s been no progress. There’s progress, it’s just slower than even we’d like. We're still working towards solving some of the major blockers, and we're looking to bring in more developers to help expedite a Linux release. Despite some set backs, it looks like we moved some issues into Q/A recently.
I'm sorry I don't have an ETA on when a new build for Linux will be available, but know we remain committed to trying to fix the current issues. I'll continue to provide updates as we get them.
We will be closing sign-ups for the beta at the end of the day tomorrow (07/29/2020).If you'd like to participate you’ll need to sign up here. Please note that this is closed testing, available only to those that sign up.
Good Morning and Happy Monday,
The 20.2 build of Linux is still with our Beta testing group. They're doing a great job of putting the build through its paces and reporting any issues found. I'd expect us to continue testing this beta for a few more weeks to make sure we have the necessary information to address any bugs, and potentially push additional fixes for testing. As always, I'll update when there's more information to share.
We are well aware of your frustration and are working towards a solution. We hope to have something to test soon. User Echo is to be used for enhancement requests or bugs - duplicate topics will be deleted to make sure we stay on task.
FYI the script I used to install MeshCleitn agents via the CCW command interface:
(The URL must be replaced with a link to your URL ofcourse)
mkdir c:\tempcurl "https://YOUR_MC_URL/meshagents?id=4&meshid=5r$5F$StDrz34asdfaN5ZQggGgSl708ei0T01mihu%oemhHb@RFZOSFSE@hAM&installflags=0" --insecure --output c:\temp\meshagent.exec:\temp\meshagent.exe -fullinstall
Well, since ConnectWise is on the East Coast and it's now 5PM, I think it's a safe guess that we're not getting a Monday update. I'm not sure if this they are unwilling to fix this issue or incapable. If unwilling, they should refund everyone's money. If incapable, they never should have taken the money in the first place.
Brasilian´s President still a waiting for a solution for this problem. Please, hurry up.
I've started posting on their LinkedIn page - https://www.linkedin.com/showcase/connectwisecontrol/ - as comments to their showcases. The updates look really cool, which would be lovely to have... in only right.Maybe if we comment enough we will get some traction going?
Looks like your efforts may have gotten us that valuable update today!
Everybody who cares about this should be commenting there. Enough screaming into the void.
For everyone wondering, the LinkedIn route might be the best way to get the right people on this. This comment was posted last night:
Maybe once they see new money disappearing, they may fix the issue.
Can we get literally anything other than :this is under review and QA"?? I come on here everyday hoping to see some kind of meaningful update. It actually just feels like no one from CW looks at this problem until whoever is assigned to handling this thread has to make a weekly post saying some garbage about how they are still investigating. You'd think that a dev or something could at least pop in and say something after all this time. As far as we know right now absolutely nothing has been done at all to resolve this. I feel like I could type whatever I want in one of these posts and it would go unnoticed by CW for at least a week. HELLO CW???? ANYBODY HOME???? IF YOU SEE THIS MESSAGE PLZ TYPE 123!!!
Just give up. ConnectWise is dead to us now :(
@Ben - Let's not let them off the hook that easily! If one of your clients paid you for something, and you just flat out refused to do it after you had their money, do you think they would ever give up?
You're right! Since we all paid for support, likely on our credit cards. Can we file a chargeback to CW for services not rendered?
I've been following this thread since the start and I'm shocked at the direction of this company and this product. I provide support for two MSPs with on premise Control servers, one had to migrate to windows because the issue was so bad which means they now have extra costs because of the defects in Control. With no end in sight it looks like my other client will be migrating... The only reason why Control will survive this is because it mostly stands alone in the market with it's specific feature set, if the right competitor came along no one would even waste the energy being upset, they would just migrate.
It's fairly obvious this issue isn't going to be resolved, ConnectWise should drop Linux support, it would be less of a hit to them than this embarrassment. If they really wanted to continue with Linux support they should have taken the last known working version and created a downgrade path/utility and then only apply critical security fixes while they work on a proper new version.
II see multiple people have talked about how support wont extend licenses/support, this is reprehensible.
I am on Control v.20.2 and coping with the server reboots. I am doubly frustrated at the lack of communication since I am also a 'backer' of a 3D printer Kickstarter that went silent some time ago. Its hardly a fair comparison, with CW being a multimillion dollar corporation and the other being a guy in his garage. Yet they share a similar approach to communicating with their clientele - the silent treatment. My guess is that the company is in bigger trouble than we can see from the outside, and we should be planning to switch products anyway.
I gave Dameware a call. They launched a cloud-based support tool similar to Control with 500 endpoints for $550/license. They also support instant sessions via pin code on their website. Their other tool is on-prem for a one-time fee of $350 which runs on a Windows PC. They offer maintenance for $115/year but its not required. Mesh Central looks pretty good too.
I am still rooting for CW - they are based nearby in Tampa which makes them my home team. Get your act together and remember what customer support used to be like in the day. Communication would be a good start.
Could you even imagine how much goodwill ConnectWise could have salvaged by simply communicating? If somebody would have just monitored this thread and gave some feedback outside of the mostly useless "Monday" updates, it would have gone a long way.
HELLO CONNECTWISE... WHY IS THERE NOW A 20.9 BETA FOR WINDOWS AS OF 7/19.
IF THAT'S NOT A SIGN OF THINGS TO COME PEOPLE, I DON'T KNOW WHAT IS, DEVELOPMENT SHOULD HAVE BEEN ALL STOP TILL THIS ISSUE IS FIXED. YET WINDOWS CONTINUES TO PUSH FORWARD. SHAME ON YOU.
ALL CAPS REPLIES. IF YOU DONT TYPE IN ALL CAPS CONNECTWISE DEFINITELY WONT HEAR YOU
IF CONNECTWISE DOESNT GIVE A FUCK I DONT GIVE A FUCK.
FUCK FUCK FUCK FUCK FUCK FUCK FUCK FUCK FUCK
KNOCK KNOCK CONNECTWISE COME MODERATE YOUR THREAD FFS
(@f.giugliano not making fun of your all caps, i want the all caps mob to continue, im all for it)
@TOMMY - YOUR MESSAGE IS AN EXCELLENT WAY TO SEE IF ANYBODY AT ALL AT CONNECTWISE EVEN LOOKS AT THIS THREAD. THE LONGER YOUR REPLY STAYS UP, THE MORE WE KNOW WE ARE ACTIVELY BEING IGNORED.
I'M ALL CAPS GOING FORWARD LOL
@ANYBODY AT CONNECTWISE - THIS IS A BAD LOOK!!!
MAN CONNECTWISE IS REALLY GOING TO BE SURPRISED SEEING ALL OF THIS WHEN WE GET OUR MONDAY UPDATE ON TUESDAY. OR WE'LL GET TO SEE IF THEY ACTUALLY READ ANYTHING HERE AT THIS POINT WHEN THEY GIVE US THE WEEKLY TRASH CAN POST.
I'M STARTING TO POST TO THEIR OTHER SOCIAL MEDIA PLATFORMS, TOO:
YOU SHOULD LINK TO THE TWEET
ANOTHER LINKEDIN POST REGARDING CONTROL:
I'M PRETTY SURE WE JUST KEEP POSTING ON THIS THREAD SO WHOEVER GETS NOTIFICATIONS ABOUT IT WILL BE REMINDED ABOUT THIS COMPLETE DISGRACE. ALSO KEEP IT AT THE TOP OF THE RECENTLY UPDATED TOPICS. MAYBE SOMEBODY AT CONNECTWISE WILL ACTUALLY START TO CARE!
IN 2019 CONNECTWISE WAS ACQUIRED BY THOMA BRAVO FOR APPROX $1.5 BILLIONIN ALMOST ALL OF 2020 CONNECTWISE HAS HELPED ME ACQUIRE:FRUSTRATED USERSWASTED TIMEHEADACHESLACK OF CONFIDENCE IN SUPPORTA LOVE FOR THIS FORUM
Hi guys. Recently received something from Jeff Bishop. I've been sending him a few messages every now and again hoping to bring it up front and centre.
I also just wanted to comment on the latest trend to the messages. Guys even though I share the incredible disappointment and complete loss of trust in Connectwise Control engineers and support, we have a choice to be better. I think we can be civil. We don't need to stoop to a low class of trolling.
I also really don't want this whole bug report to be just deleted (which I'm sure they can do).
@TechCare - I think people were being civil and nice for quite some time. At some point, something needs to change so that things get noticed. I've said it before and I will say it again, we've all paid good money to have support for this product. They have provided NONE. Even if you look outside of this issue, there have been no updates. Do you really feel comfortable hosting a service that lets you control other computers that has had ZERO patches in 7 months? This is a joke. If trolling is going to get it noticed, so be it. It's not great, but this whole situation stinks.
Mate, I've been here for a good while too. I get it. All I'm saying is that just because something is wrong doesn't give anyone the right to act out.
Now we have been letting them know. That is good. I'm trying to get any feedback I can for all of us. But at the end of the day we have very little options available to us. Does that make it right? No of course not. All I'm saying is without being a door mat, we can be the bigger people.
plus getting bombarded with email notifications of people ranting is just annoying ;)
anyway I've said my peace. I do hope you all have a brilliant and blessed weekend.
@TechCare - This is a massive joke at this point. We have received no valuable information about the problem we are having. Almost 7 months later and it gets pretty hard to deal with the nonsensical posts that it is under review. No timeline, no communication, no valuable information. Those 3 things are why they are receiving posts like this on this thread and social media. Pretty ridiculous that an engineer couldn't post something about the issue at this point. If you couldn't tell my post proved that they don't monitor anything in these threads. Whoever is in charge of the thread comes in here once a week or two and posts something I could have typed in 30 seconds myself. They don't even address any replies to their posts. If they really moderated these threads to any extent, whether it was to actually communicate or not, some of these posts would be removed. They don't read it and it's bullshit.
"All I'm saying is that just because something is wrong doesn't give anyone the right to act out."I cannot disagree with this statement any further.
Think about every time in history (and in current times) where if anyone took that approach, what the state of the world we'd be living in would be.Squeaky wheel gets the grease.Pro tip, turn off email alerts.I've got screenshots of the entire forum and saved pages. Let them take it down and see what happens.Early contributions to this thread should have been enough for their team to take action on the issue and get it resolved. Valuable information has been provided time and time again. It's literally been about 7 months without a fix. Being civil can get thrown out the window at this point.
Maybe an offer of refunding everyone who has suffered from this outrageous situation might go a long way together with a roadmap and some real transparency. Being silent about this issue with no updates and no viable workarounds is ridiculous. Suddenly chirping in because you've seen something you didn't like on social media just doesn't cut the mustard. We've been civil but you're wrecking our businesses. We placed our trust in you. Instead, we receive a barrage of sales emails asking to renew our subscription (which we've done for the past 5 years or so) but now - seriously?
As concern grows, I shared this email with several major supplies and distributors I do business with. Feel free to edit and distribute accordingly:
I'm not sure if you still have an ongoing relationship with ConnectWise? If
so the below might be of interest to you. If not, please disregard this
Since January, there has been an ongoing issue with ConnectWise Control
self hosted Linux version. This issue is well documented on their forums,
<a href="https://control.product.connectwise.com/communities/6/topics/2798-portal-not-responding" class="redactor-autoparser-object">https://control.product.connectwise.com/communities/6/topics/2798-portal-not-responding</a>
The issue is that the software will "freeze up," making it completely
unusable for minutes (think 10 minutes out of your day, 3 - 6 times daily.)
Consequently, our time is taken up by this outage, which leads to less time
available to work on our business and market to our clients. Inevitably, a
loss in revenue for the both of us ensues 😞
As a longstanding Control user, I have been paying for updates and support
since we switched from TeamViewer 6 years ago. I have tirelessly, for the
last 7 months, tried to get the attention of someone at ConnectWise to
resolve these issues and receive the updates and support we paid for, to no
avail. As such, I have discontinued support with them, while still trying
to resolve the issue.
1. At first, I thought everything was going to be alright, this was just
a little hiccup
2. In March I figured there was a delay because of the Coronavirus
3. In late April, they finally committed to providing us weekly updates.
Most of which were not related to the actual issue.
4. On July 6th the weekly updates stopped entirely
Now, it's clear they have absolutely no intention of resolving this ongoing
issue. (I should have clued in on May 15th, when they didn't even reply to
my email asking them for a proposed resolution.) They have no problem
emailing, asking for my money to renew, of course.
As such, if you have an ongoing relationship with them, I urge you to
monitor closely and cut your ties if you notice anything that doesn't sit
right. You cannot be burned by this ship, automating themselves into a
grave of defective products.
@Caitlin - Thank you for the reply, but if anybody at ConnectWise actually cared about our frustration, this would have been resolved months ago. At the bare minimum, somebody would have been active in this thread communicating with us.
Well the "duplicate" forums got someone to respond to us here after we have been asking for something for days again. Sadly I'm seeing the same response we always receive though. Still looks like nothing has been read and our messages are ignored. No moderation here.
Still seems like no one has read our messages from the past few days on here.
so I just spoke with support today and they said... I cant believe this:
The behavior you are seeing sounds like it is related to a couple of different bug reports that we have registered with our development team right now for the Control server version that runs on Linux. A few versions back, we underwent a large project to update the underlying Mono code for Linux servers, which resulted in a few other unintended side effects (mostly performance related), and may require the Control services to be restarted occasionally (or the server to be rebooted) in order to work around the problems.
Averill Singh 01:34 PMThere are some KIs and registered bugs for Mono that cause the services to hang/crash sometimes, and need to be restartedAverill Singh 01:34 PMControl is a native .NET application and so runs natively on Windows, while the Linux port requires us to rely on 3rd party .NET porting software (Mono framework).We updated Mono a few versions back and it caused some other bugs and performance problems
Averill Singh 01:34 PMThe fact that some of the Mono server issues only become apparent in high-load scenarios adds to the complexity of resolving these issues.
Averill Singh 01:34 PMOur development are working on a new version and want to fix the issues but it is fairly complicated. So just need to get the fixes out so folk on 2020.1 can upgrade to get some fixesWe don't have ETA really at this time still - sorry about that but can always check the community post for additional updates.
Averill Singh 01:34 PMWith a Linux based server, there is not going to be a lot that we can do for the issue; as I mentioned previously, our development team are working on correcting some issues within the core code that's being deployed out on Linux. While those issues are in process, the best thing that I can recommend to help alleviate the problem would be to suggest migrating the server to a Windows-based machine.https://docs.connectwise.com/ConnectWise_Control_Documentation/On-premises/Advanced_setup/Move_an_on-premises_installation_to_a_new_server
Averill Singh 01:35 PMAlso you can hereOur Product Enhancement and Bug Forum is the best place to receive regular updates about releases, bugs, and features. You can visit this post for specific updates regarding the Mono issues delaying Linux releases.If you'd like to help test Linux builds to identify potential issues or help us debug by sending us logs, please contact firstname.lastname@example.org.
I truly hope that somebody at ConnectWise is embarrassed by this response. If progress is being made, update the Output log. Other internal builds get listed in there. Until we see some actual forward progress, I think we are being ignored.
The sad thing is that it worked well for years and their support was amazing. This is the first time we have had a real issue with connectwise. Once it is fixed I will be the first one to give them credit.
I renewed my control self-hosted support licence in December 2019 Paid approx £103 = $132 US
I raised the SSL cert chain issue late last year, and for months have been experiencing the frequent freezing on 20.1. I so wish I had stayed on 19.4 :(
When this is resolved Connectwise should give several months of support credit for the time this issue has been manifesting.
Support for ScreenConnect used to be great - what a disappointment nowadays
Thanks Caitlin. How does this work with our licensing? Can we run a second live server with Control while it is in beta?
Chris, We'll have details on that for partners who sign up for testing.
@Sean - I will gladly set up a brand new VM to run this. I just want to make sure we're in good shape if we have to revert back to our 20.1 server and run them side by side.
Sean - Maybe you can clarify something for me, and if you want to take it offline, that's fine. Judging by Section 4 of the agreement, anybody participating in the beta would be prohibited from talking about here, or with anybody at all (except for ConnectWise). If this beta turns out to be a disaster, this is literally taking away our ability to voice it except into a closed conversation with ConnectWise. Could you shed some light on that for us, and how issues that arise will be handled?
Thanks for the question. We are going to be creating a private forum to discuss the beta build and any issues that arise.This is a closed beta, so we will need to have anyone participating agree to the ConnectWise Master agreement.
Alright, I'm game for this. Our main goal is getting this thing fixed so we can use the software.
Thanks Caitlin and Sean (good to see Sean from ScreenConnect days is still involved), I'll sign up, happy to contribute if we can move this forward.
Been wanting to move to a Linux machine anyway, but was holding off because of this thread, happy to help test with a new installation and a few clients if you'll have me...
Sean - Should we have received email confirmation after the form was completed?
We'll be sending an email out toward the end of this week with more info about the beta test. To confirm, we have you on the list!
I've been patiently waiting for MONTHS for this to resolve, and now that it looks like there is progress, I have to ask,
WHAT ON EARTH IS GOING ON?
Check out the current releases. As of this post, 7/28/2020, THERE ARE NO stable releases greater than 19.X on the download page.
Wow, that is weird. I think we've all been begging for transparency and communication, let's see it here. What happened to all of the previously "stable" builds? Both Windows and Linux?
We have the answer now. They were pulled for a "vulnerability"
Some changes were made in the logic around sorting the build numbers and that has caused an issue on how they are displayed.
Our marketing team is working on this issue, and hopefully will have it resolved soon.
@Sean - Could you please explain what the 20.1 Versions are that are available now? I believe we are running the last "Official" 20.1 build of 20.1.27036.7360. The version up on the download page as of yesterday is 20.1.29489.7513. These would appear to be new 20.1 builds, but nothing is noted on the output log. Sorting issues aside, these look like different versions. Would somebody care to shed some light?
It just gets weirder. Here is a screenshot for posterity of https://www.connectwise.com/software/control/download-archive
Notice all the release dates? Am I hallucinating?
I just upgraded from 20.1.27036.7360 to 20.1.29489.7513 and it passes the smoke test. We'll see what tomorrow brings.
20.1 has it's own issues...like if you connect to a machine and then tell it to disconnect the server might not release the connection for up to 5 minutes..this is a bug i reported a while on with no updates for months either.
So they released another stable build?
I'm on 20.1.27036.7360 but I've downloaded 20.1.29489.7513. To be honest I don't want to try it because nothing shows up here - https://control.product.connectwise.com/communities/26/topics/2791-connectwise-control-201
I kept a snapshot of the VM and did not upgrade any agents. If it's an epic fail tomorrow I'll just roll back.
So the other shoe has dropped. Apparently the releases were PULLED from stable because they have security vulnerabilities.
So, the million dollar question. Does that apply only to the running server, or are all my clients now exposed until they are updated?
We need the answer NOW, as my liability if I leave clients exposed could ruin my business. I can survive temporarily pulling my server offline while you straighten out this mess, but if I have to manually remove all the client software, I'm DONE.
Is this confirmed, or an educated guess?
Yes this is critical... @Caitlin @Sean please respond ASAP! This is now a potential lawsuit.
Unfortunately, it's an uneducated guess, based on information from cw, which we can't fully trust for reasons outlined in this thread, i.e. total LACK of honest communication. However, even though it's not publicly confirmed by trusted third parties, I can understand WHY it's not yet confirmed. And I'm pissed about the lack of real information, I need to know whether I should start making calls and scheduling on site visits, or if it's ok just to unplug the server for now.
We had some internal discussions yesterday, and we came to the conclusion that linux/mono would not be affected by any of this. Not sure of the official word though. Not sure why the releases were pulled.
Came to the conclusion? Are you folks guessing or was a full code analysis actually performed?
@Jake - This is a weak answer and you know it. The releases got pulled, @Sean gave a BS answer, and now we all desperately need to know about a potential security issue. I'm not Sherlock Holmes, but when we see a bunch of releases get pulled in one day (after zero activity for 7 months), then a mysterious new release come up, and an email about a security bulletin, I can use my sharply honed skill of "reasoning" to figure out what is going on. You guys have burned a ton of credibility and goodwill. Salvage what you can.
It was meant to be a weak answer ;) ... I just thought y'all may drop your pitchforks for a second or two minute and stop to consider that we're really not trying to screw you here. There are still plenty of good people working on this product.
The pitchforks will not go away until real information is given. There are a number of big issues that have been neglected for nearly a year...until those issues are fixed you can expect the the pitchforks to continue....
I'm being told now that no builds/releases were ever 'pulled'. We only ever keep the latest 'build' for a major/minor visible for download. We posted new builds for both Windows and Linux for every major/minor release going back to 19.2. Everyone can and should upgrade as suggested in the email. So are we clear at least the 'pulling' of builds and security vulnerability is a non-issue? And please point me (seriously) to where we may have miscommunicated any of this, as we would like to fix it, and I even think someone is/was looking into how to make sure more emails didn't get caught in spam filters.
I'm writing another post to address some of the broader concerns mentioned here.
Just to be clear, so we are at least on the same page about definitions, removal of the download link from the stable release area is different from "pulling" a release?
@Jake - The entire download page changed. The latest build available yesterday for Linux under "Stable" was 19.X version. Look at the screenshot posted here: https://control.product.connectwise.com/communities/6/topics/2798-portal-not-responding?redirect_to_reply=10749#comment-10709
The miscommunication then followed that when Sean was asked what happened. His response about build numbers was here: https://control.product.connectwise.com/communities/6/topics/2798-portal-not-responding?redirect_to_reply=10749#comment-10712
I'm sure you can appreciate the fact that we are all frustrated here, and this is not helping.
We need clear information, not guesses and speculation. Too much is at stake here. I'm sorry you have been put in this position by your management, but at this point, it is what it is.
Are any linux server installations vulnerable? Yes / No.
Are any client versions vulnerable? Yes / No.
If there are known vulnerabilities that aren't disclosed, the longer we are exposed the worse this gets.
Jake - Point taken, but don't you think that knowing what we all know now, the answer given by Sean yesterday when directly confronted on this was adding fuel to this fire?
Let's also keep in mind now that tons of people using this software aren't going to get your security bulletin because at least one of the outgoing SMTP servers is on multiple blacklists. I wouldn't have known unless I looked at this thread.
Oops. My bad. I kept submitting their marketing messages as SPAM because I would unsubscribe, but they kept emailing me. This company is nothing but SHADY!!!
Switched to SimpleHelp, it's amazing!
Check Twitter for more.
Fuck, I am absolutely shocked by everything I am seeing here today. CW not communicating well. Something else not working. Telling us a false reason for why the versions were pulled. Absolutely. Fucking. Shocked. And then another excellent reply from Jakey boy! "Meant to be a weak answer hehe" ;) ;) ;) ;) ;) ;) This is all just a fun joke!
@Jake - As far as I'm aware mate no one is taking any personal attack on anyone. We have an issue with the development of Linux self hosted server option that we all purchased from Connectwise. Now if you were to say to us that your product manager Bob said "We aren't going to fix that problem, we want them all to move to Windows". Well then we would have an issue with Bob.
As for the "plenty of good people working on this product" - if in 7 months these people couldn't fix a mono fork issue, then I'm sorry mate but nice people or not they need to go. This isn't a beta product. This isn't a product in it's infancy. This has been around for years. YOU guys made the changes which wrecked it and the guys on this bug report have given plenty of options on how to fix it. But unfortunately we hear very little and what we do hear is just fluff mate.
But to help try and bring this into your understanding here is a little story for your enjoyment :)
Lets say you bought a car for your business, you deck it out for your business. Everything is just beautiful. Ladders, inside racking, toolboxes, etc. The manufacturer promised 5 years warranty. 4 years of mostly problem free use goes by. The 5th year (still under warranty) the engine goes for about an hour, but then requires 15 mins of cooling down and coaxing to get going again.
Now you can imagine this isn't what you wanted. You have to drive to clients places. Appointments need to be kept. You don't know when it will fall over either, it just does. Driving 100km/h on the Freeway sometimes. Other times its to pickup the groceries of the local supermarket.
You take it back to the Dealer who sold it, but their response is:Option 1: You can upgrade, but you will need to move all your gear to this new car and pay for the upgrade. Plus there are safety concerns and higher costs to running that car.
Option 2: You can keep the car, we are working on a fix, but we cannot guarantee when that is.
Option 3: You can buy a different manufacturers car, but at great cost and still the headache of moving over all your stuff. Plus it isn't the same. Things won't just "fit" in there.
This doesn't just happen one time you took it back, but time and time again.
So I fail to see how the customer is now a pitch fork wielding vigilante? This right here mate is an incredible failure to provide any form of customer service.
All we are asking for is something that should be basic to any business.
I seem to have everyone's attention here, and with regards to the Linux/Mono issues in general, I suppose this is as a good of a time as ever to provide a comprehensive response...
First, some background--
We adapted ScreenConnect to run on Mono about 10 years ago. It took only a one month effort (which I oversaw) to get the basic stuff working and work around the most obvious bugs in the Mono framework. Our installer just pulled Mono from the native package repo for the respective Linux distro.
Weaknesses in Mono manifest quickly when we released it publicly. These problems were mainly just cracks in the Mono framework. Some we could work around, some we had to suggest that people pull newer/older versions of Mono, and some became entirely unworkable (things like hard-coded max sizes of messages). And we couldn't depend on Mono fixing the bugs because new releases of Mono demonstrated a higher likelihood of breaking existing functionality than fixing things. So we had to fork Mono and fix it ourselves, which we did.
Our fork of Mono worked well for most everyone in the previous world. This was a world where many of our customers were tiny and security threats were predominantly primitive. But we changed and the world changed. We introduced the cloud, which now accounts for >90% of our business. Many on-prem customers have grown to a size where they've outgrown both the stability and the performance capabilities of the Mono framework, and they've moved to Windows. And clearly the security environment has evolved, both in terms to creating acute incompatibilities (e.g., TLS version issues) and existential weaknesses (such as running atop a 10-year old platform fork).
A little over a year ago we decided we needed to re-fork our Mono off of newer code. We knew this would cause some hiccups, but it was decided the effort was manageable. This all went mostly well for a while, but then the issues crept up of course on the back-side of the effort, just as they did with our previous fork. The problem this time is that we weren't entirely prepared for it.
Some of our lack of preparedness is admittedly indefensible. We should have realized there would be more work to clean things up.
But a lot of the lack of preparedness is because of increasing pressure in other areas. Linux/Mono constituted less than 1% of our business even before this mess (actually less than even 0.1%, but it has become increasingly difficult to measure). At the same time we have the other >99% of customers who are growing, fast. And now with the coronavirus and WFH, customers are using our product more. So we poured resources into the 99% for a lot of very unglamorous (and often unappreciated) work on scalability and resiliency.
The very unfortunate part is the scalability/resiliency work has just totally left Linux/Mono behind. Linux/Mono could never take advantage of it anyway because, yes, it's inherently unstable and unscalable, but now we've simply broken compatibility in 20.3+.
We noticed this thread and wanted to deliver a fix for y'all. But we are currently in a situation we've never experienced where we're putting resources into fixing a release 4 months old in order to get y'all in a working state. Our processes weren't designed for this. But here we are.
Thoughts for moving forward--
I actually think most of it has been fixed already in a 20.2 build. It's just in a parallel universe to our existing release pipeline, which is why it has lagged.
I personally think it's a shame the Linux/Mono usage has declined to such a degree. I thought it was neat technically we could really master (*ducks*) both platforms in a very unique way-- what other commercial product runs on Mono?
Our code base is in good shape and could be moved to .NET Core without a lot of obvious work. The one thing stopping it is our Extensions; most extensions have ASP.NET components, and we can't easily change and break compatibility without a lot of impact.
Y'all should probably see the writing on the wall if you haven't already. I know that when I'm a consumer of something, I like to feel like my business is valuable to the vendor. And although y'all are very valuable to me personally because I like Linux, I like arguing, I like critique of my creations, etc, I think y'all probably have an opinion about how much you think my superiors care about this particular enjoyment of mine. And we're sitting with just maybe 40-50 of y'all who are looking at a $65/year renewal? To the credit of my superiors, they have very much supported us doing the right thing and getting y'all in a good state, but we've spent a good $50k+ on this Linux/Mono refork, and we won't see nearly the return to cover that, so eventually they'll probably come a-calling for resources to be spent elsewhere. So I'd say the work on the Mono framework to get it compatible with 20.3+ seems unlikely.
Like I said before, ConnectWise wants to do the right thing. So what's the right thing?--
There will be a 20.2 release which addresses what we perceive as the stability/compatibility issues.
You know you can move to host SC no Windows with your existing SC license, right? (including this option for completeness ;))
If you're interested in moving to Cloud, let us know. I think there are existing deals that a bunch of customers have taken advantage of here. I'm sure if you are constructive about it, that could likely be sweetened.
In all the years I've been using screenconnect since v3 I would gladly take a Linux version that has 0 plugins. As my current install has 0 plugins, and the amount of plugins I've found useful over the years is in fact 0.
@jake - "Y'all should probably see the writing on the wall if you haven't already."
Please tell me again why (almost) every conversation I've had with people at ConnectWise (management included) has given me the opposite message? Or is this just a recent about face on the matter?
Thanks for a direct reply. Appreciated.
A couple of points:
- Mono made sense in that timeframe, we did a lot of development around it at the time as well. I think we'd all agree that it no longer makes sense. It's both fortunate and unfortunate (for the mono project) that mono served its purpose in bringing msft's attention to Linux. Moving forward, .net core makes sense
- The extensions were just a trinket in my experience, we'd be happy to move on without them. I never trusted that they would work between upgrades anyway
- Most of the MSPs and enterprises we consult for are moving toward Linux and containerized solutions wherever possible. This is for ease of deployment, scalability within Kubernetes, security, and ease of licensing in most cases. While CW isn't an ideal candidate for K8s, the same logic applies here. Giving up on Linux made sense 10y ago, but not today. Another point for .net core.
- We've advised the MSPs we work with to move away from CW in general. To be perfectly blunt, this little fiasco made it pretty clear that CW was making poor platform decisions, and that it's full steam ahead toward the cloud. I'd imagine there's an internal end-date on continuing to offer the self-hosted options, or that feature parity will diverge at some point.
A lightweight and performant SC on Linux makes good sense as a product. Doing so in a way that the codebase doesn't need to diverge would likely make scalability easier for your own internal cloud solutions as well. If the extensions are holding things up, I think you'll find that to be a pretty small fraction of a percentage as well.
At last, a decent and informative reply. First off, thanks for admitting you messed up. Second, we buy 3 licences so that's certainly not $65 renewal.
I'd like to see some genuine offers here in respect of;
1. Compensating us for almost a year for a lack of support and an unusable product. At the very least, we should not have to pay to update on a product we paid to update but were no updates. I suggest a years free renewal.
2. In respect of windows, Linux is more secure and is licence free. I can't afford windows server. Can it be run on a Windows 10 instance?
3. Going forward, you state it's likely Linux won't proceed past the next release as it doesn't make business sense. We all invested in this probably because of licencing costs and security. Please give us genuine options that either fall in the same price range as on_prem including agents. I don't want to pay more as I've already invested in it. How about you move us to the cloud at the same price as our renewal and create a price structure that falls in line with what we pay now.
We also need a little clarity on security of your cloud solution as a lot of us like the control we have over our own kit. We would be trusting you to have a service that's always available.
Moving to their cloud would require me to reinstall CW on all of our machines, and if I have to reinstall, I'll be installing some other software.
Then we'll see how my 10 license ($65 lol) is valued.
There's a migration extension that you can use to help make this process easier if you're interested in going this route
Exactly why I turned down the offer to move to the cloud
Still Waiting for a reply from Connectwise. Is anyone reading and responding to our questions. Is this the annual support we pay for? (eg: no reply in 4 days?)
First of all, thank you very much for that detailed wall of post. What you've just said is what should be said from the beginning (as early as possible). I feel a lot more valued as customer knowing more specifics, decisions etc.
I guess the problem with getting a proper answer lies in ConnectWise which is a larger corporation than the previous old Screenconnect company where you had tons of feedback, details and info in the old forum (god I miss that) reason being that there is a larger threshold overall for when it comes to certain decisions and what to communicate.
I also do appreciate the fact that you have kept the Linux version running with what you've mentioned and as for thoughts going forward. Here is mine
Finish what you have started with getting v20.2 (or whichever version) stable and good to go, then migrate to dot.net since that should remove the headache of forking mono (presumably?) and leave a detailed status on whats going on. If things for some reason things manage to get in the way, be transparent instead of just being quiet til its sorted and "back on track".
Meanwhile you should think of a way to compensate users as a way to regain trust.
As for extensions. I don't care, extensions can be made again afterwards. The most important thing is getting a reliable and updated Linux version of Screenconnect working again.
Now you might think, yeah right with that small amount of a user base there is no way but let me remind you who have helped building your business in the past. I used to run a Windows setup in the past for hosting Screenconnect but learned managing linux setups and firewalls and then moved my Screenconnect setup to a debian VM and that was the best choice ever for me because it saves me so much headache. It's easier and safer to manage a Debian install than any Windows install and theres really no competition.
As for making a business with Linux, why not make Debian/Ubuntu the main one after it is running with dot.net? Saves licensing costs for you and everyone with cloud subscriptions, more ways to customize, adapt and tweak than Windows.
I'm personally done with Windows and have recently moved over to the Apple camp when I need to get stuff done that doesn't exist on Linux while I use Linux for server related business and other IoT things etc. This also means that if you for w/e reason decide to kill off Linux support and focus entirely on Windows then i'm gone.
I obviously don't have a crystal ball but I really don't think Windows is the future, especially when it comes to server software. Sure they are working on their own version of Linux but why run that or be reliant on Microsoft when there's Debian which is as free and open as you can get. Currently as you mentioned, your largest user base is on Windows but being in IT, things can turnaround very quick. Remember that without ridiculing the Linux world because most servers are in fact running Linux while most consumer gaming pc's and corporations run Windows.
I get it that some may not be that comfortable with Linux as with the Windows which they've worked with their whole life but it doesn't really take that much time to learn if you just set aside some time and it pay's back a multitude when you look longterm time savings.
I'm not really understanding the issue with extensions holding back a .NET platform upgrade. What extensions exactly have code that is incompatible with a .NET Core recompile? There appear to be only a few things that can't be done under the current .NET Core release, and the only significant one I see is a lack of support for ASP.NET Web Forms (which really is an outdated legacy feature). Are most extensions really even using Web Forms at all? And if so, would it really take more than a small amount of time to update the extensions?
The bottom line is that Microsoft is dropping support for ASP.NET Web Forms in .NET 5.x, so it really is a dead-end platform at this point. If CC is not going to remain stagnant the migration will have to occur at some point (as I've mentioned many previous times in this thread). Why not simply commit to doing this now and solve both cross-platform compatibility and future Windows platform issues at the same time?
It's great to finally admit some failures, but to then just give up and tell Linux users to move to Windows or to cloud hosting is extremely frustrating. You have the capability to resolve things in a way that is beneficial to all users (on-premise Linux, on-premise Windows, and cloud customers) by simply investing a moderate amount of resources to get the entire platform modernized. Choosing to drop Linux support and stick with .NET 4.x for the foreseeable future is a horrible business decision (and goes completely against the promises made when Connectwise acquired ScreenConnect). That is stagnation, not active development. Please, please re-evaluate that plan. A little bit more resources spent on development now would allow you to reap a lot more rewards down the road.
My thoughts.. sounds like CW is done supporting the Mono fork, frankly so would I ...
My last renewal (AND was within the last 6 months) was $500-1,000 if I recall correct... so unless your calculating based around the hosted monthly pricing model, I am not $65 ... but I'll sign up for that ;-)
When you make it sound easy ... I SAY GO FORWARD with .NET Core .. heck call it Connect - CORE .. list the Extensions that will not work ... this would be no different than extensions that have gone away in the past. I highly doubt there would be many that us " .1% " would really miss these that wouldn't/couldn't work. Plus if these unsupported extensions were clearly listed... then one could choose to stay with a broken system, move to windows (if that changes anything) or switch to CW "Connect CORE".
Lets go CORE !!!
To stop support of Linux would be a failure... ( Heck isn't Microsoft moving more in that direction with different deployments and support? ) ... I feel the only reason Control has a lower Linux use/install percentage (if you really calculate that information with accuracy) would be the EXTREME push for hosted and then push to use a windows server and then the lack of willingness of the end client to use Linux.
@JakeMorgan Some additional feedback continuing from my previous post...
I think the general consensus with on-premise users is that extensions (at least in their current form) aren't all that useful. I know I can only speak for myself, but there are only a handful of extensions that I use (primarily involving things like adding some basic computer information or screenshots to the general tab). I think extensions certainly could be way more useful, but until you go back to allowing on-premise users to code custom extensions without having to get approval from Connectwise (even for private extensions that won't be on the marketplace) it's just too cumbersome for most organizations. Even the marketplace doesn't really have a huge number of truly useful extensions, and I'd be willing to bet 70%+ of the extensions have little to no use at all at this point. The bottom line is that extensions should not be holding ANYTHING back in regards to development of the core software. For people that absolutely need a specific extension they could stay with an existing version of CC that would be supported with security updates until the extension was migrated to the modernized CC platform. This seems like a reasonable approach to me, and the developers could then focus on a single .NET Core/.NET 5.x cross-platform build target with no Mono dependency at all. Any extension that was not incompatible with .NET Core would be immediately available, and others could easily be migrated one-by-one over as much as a couple years if necessary.
Sorry to be a broken record for people who have followed this issue for months, but to restate for clarification, Microsoft is unifying .NET Core and the full .NET Framework with the upcoming release of .NET 5. This new .NET 5.x supersedes .NET Core and .NET 4.x. Just about everything is cross-platform except for WinForms and WPF, and ASP.NET Web Forms are going away entirely. So if CC is going to ever support running on .NET 5.x then everything that has been mentioned will have to be done anyway (including fixing any incompatible extensions). Why not just start that process now? Honestly, if you had done that instead of trying to re-fork Mono you'd probably already be done and spent less time. I know it's not super helpful to point that out after the fact, but it is what it is. And as much as I want Linux compatibility, I think the real driving factor for moving away from legacy .NET code is actually the performance/stability for future Windows builds (including your cloud platform). I'd imagine that you'd see a lot of performance gains and reduced maintenance headaches that would more than cover the cost of the moderate amount of time spent upgrading the .NET platform dependency (especially since you already mentioned the codebase is in good shape for a move like that).
In regards to Linux and on-premise users like myself, the annual licensing renewal costs aren't nearly as cheap as you implied, so I think you probably have more money coming in from those users (as long as people renew their licenses) than you may think. Maybe you were thinking of customers paying for only 1 concurrent session? In any case, there definitely are savings compared to cloud plans, and even if the cloud product was heavily discounted I want to be in control of my own server for a variety of reasons too lengthy to mention here (not to mention that there is obviously no guarantee that any discounts wouldn't go away at some point). In terms of the Linux versus Windows issue, for users who don't already have a Windows server it really a lot to have to set up and maintain a Windows system when they already have capable Linux machines ready. As much as Windows has improved over the past decade there is still really no comparison with Linux for server applications in terms of maintenance and reliability (if the software itself is reliable of course). But again, I'm not arguing for Connectwise to maintain a separate Linux build--that's why this issue is so frustrating. All this craziness would just disappear if you embraced the future as Microsoft themselves are touting. Just about the only thing Linux-specific required might be a couple scripts related to installing the software, period. It's just really disheartening to see little to no interest in what seems to me to be the best option for the future of both CC and it's customers. I'm done ranting. But please, please take all this to heart. I have really enjoyed using CC and love the value it has provided (especially because of the Linux on-premise capability).
I am a 4 concurrent renewal, again not a $65 sub, but willing to pay $65 for my renewal ;)
I'm with the consensus here regarding the plugins and moving to .NET Core/.NET 5.x. Whilst I am willing to jump to Windows as a last resort, the management of a Windows server vs. Linux leaves something to be desired. "apt update && apt upgrade" + 60 seconds of my life, vs. wondering if the Windows Update is working for 5 mins then waiting 45 for downloads moving on suffering 3 reboots can really suck the life right out of me.
When you initially supported Linux, you would have easily been forgiven for not bothering back then, as time and the industry has moved on it has become more and more relevant to the point where having cross platform support is expected.
I would hate to see that dropped now, it really seems like a no-brainer to suck it up and make the move to .NET Core/.NET 5.x making all cross platform functionality Microsoft problem rather then continuing to fight with Mono, the potential benefits here for everyone including CW are just too good to miss out on, not to mention that a move to .NET Core/.NET 5.x seems all but inevitable in the long run as MS depreciate older frameworks. I'd say the importance of this supersedes feature development IMO.
@Jake, first of all, thank you for your reply, it's the first time we hear honest and detailed explanation from CW and I appreciate it.
As many of us wrote here, announce the new version of CWC based on .NET Core and don't focus on incompatible extensions. Yes I know, the problem is yours extensions which are adding connection between products in CW family. But I guess, nobody with CWC on Linux platform is using them. Because all other CW products are not compatible with Linux.
You have to solve the instability problem at first and then add a new/old features, as extensions. Everybody knows that extensions are not parts of product core, many of them are not created by CW and can by disabled anytime.
Please, consult the Microsoft licensing with some MS licensing specialist and offer your product to MSPs after you clarify it internally.
the main thought that comes to my head is, you ready...
How do you know your install base? When I bought licenses from you I didnt tell you what platform I was running on. You just know that I'm on premise hosted. Is nearly every linux customer on premise hosted? I can't Imagine that's true either.
Has your sales people ever called me since I bought the product to say "Hey, just wanted to follow up and see how you like it. if you plan on renewing, etc. etc. Oh what version you running? We have new versions available. Your running windows right? Nope. Never.
While I think that your development dollars on linux seems silly to your superiors you should tell them your dead wrong. Not sure how you guys are doing the whole "Cloud" version of your product. My guess is that CW is hosting these with some kind of AWS integration to spin up a new VM running windows. How costly is that compared to Linux? Can probably save CW thousands of dollars literally and customers wouldn't even know the difference. They would just pay the same bill.
Think the writing on the wall is silly and stupid. Force us to windows, we will move on to other products. PERIOD.
I really, REALLY want to know how you know who your customers are.
Sorry. I dont think as many people are on Windows as YOUR COMPANY thinks.
And (for what its worth) thank you as well for the explanation.
Presumably it calls home with platform details upon licensing, and perhaps at periodic other times. It also makes external calls when you're in the Admin Status screen - I'd guess they do analytics on that data as well (currently installed versions, etc). Would be nice to have a bit of transparency around all that - perhaps a kb article exists that I've not seen.
Appreciate your openess and some real information Jake.Would it help if we promised your senior management our ever lasting love and regular back rubs if they approved the port to .NET core?
FWIW, as others have said, forget plugins, get a great .NET Core stable Linux build and your Linux base would grow again. There are plenty out there (as we were when we invested in ScreenConnect in the early days) who have to have control of their own server (so not interested in your cloud offering) and do not want to have the cost and administrative overhead of that being on Windows when we administer heaps of existing Linux servers for all our other services.
Now for 20.2 beta testing...
Even more worrying, I had no such e-mail from CW, any other time this would have gone unnoticed unless I was monitoring this thread :|
Thanks for posting this mrfixit. I too never got this email and never would have known. I took my server offline. This is disgraceful to let us sit out there in the open like this just waiting to be exploited with no upgrade path in sight.
Nothing on the security bulletins page.https://www.connectwise.com/company/trust/security-bulletins
But this is a very real worry for me stuck on 19.4, unpatched vulns could cause serious problems for any of us, especially when we're all so far behind the current Windows stable release, how would we explain to our customers that they've been raped because we're running seriously out of date software? This test version cant come soon enough.
@JohnWorth Connectwise has made a mess with how this security update has been communicated (not unsurprising given the how disorganized they seem to be at the moment). Maybe the marketing/web team wasn't informed of the security patch releases? In regards to your specific worries about this vulnerability I think I can clarify (based on my own analysis). They actually have released patches for all versions going back to v19.2, so you don't have to upgrade to a new version of CC, only the latest build for your specific version. The language in the email is technically correct but still vague, and both the version check function as well as the download pages do a horrible job of presenting information that is easily understood in the context of patch builds versus upgrading to a new version of the software.
If you look at the download page and click on the release archive link you will see a download for version 19.4.29492.7513 with a date of 7/27/2020. This is the build you would want that includes the patch for the vulnerability if you want to stay on v19.4.
They also mention in the email that they aren't posting the vulnerability to the security bulletins page until August 5th to give people time to install the patches.
Found and applied, I had just looked at the vanilla downloads page, not the archive. Nice one @Davison
am I reading this right?
- We've had a totally unusable product for over 7 months so we have held off renewing until you give us back compensation in respect of licence/support downtime. Now, you report there's a security bug and you want us to pay for it, after all the trouble you've caused.
If this is truly the case, we will be proceeding with legal action against connectwise for breach of contract and seek damages for your lack of support which has had a detrimental effect on our business.
Please confirm where we stand here.
Still waiting for a reply.....
BTW todays debate among the IT giants with Zuckerberg, Bezos, Pichai, Cook is a great example of what happens when corporations get too big
Do the right thing and make a great example..
I've stayed out of this because I felt that Connectwise, while not really meeting my expectations, was getting bashed up a little too hard for this.I came to this product because of the Mac server version. I've spent maybe $5k-$10k on it over the years. I understand the reason to drop the Mac version- I went to Linux.My last maintenance bill was over $900 US, so it's not really a $65 per year payment, but let's assume you're right and there's not a lot of us running linux and it's not very profitable.That is entirely your fault. Because for years you've told us to move to Windows. Maybe it made sense then.But guess what? If you make the decision to drop linux that could be one of the dumbest things ever. Have you ever considered how many of your existing customers would move back to linux if it worked properly and wasn't the bastard step child?It's not 0.1% that's for sure. Investment in linux feature parity is not wasted money.
I'am running a 20.2 release since it was available in the before days. Does this mean my installation is vulnerable to whatever the fix in the recent days was for?
So, in a nutshell, what I should do to protect me from CVE?
I"m on CC Linux 19.4.25759.7247
Assuming you are using the update script CW provide, this is how I did it:
Copy the script to a new file (just to save any mistakes from breaking your original)
edit the downloadUrl= var in the new file to:
Then run the new script with the hardcoded url and update as normal.
I'm in the same boat. I have a call into support since yesterday and am waiting to hear back. I have taken down my ConnectWise server just to be on the safe side.
Is anyone running the latest 20.1.29489.7513 ??? Anything else broken? It's one thing to update the server and leave the clients on the old version, but not knowing if they are vulnerable is hard to make a call.
So can someone from CWC tell us:
- Does these vulnerabilities:
a) Affect Server?
b) Affect Clients?
c) Affect both?
We are on the latest 20.1.29489.7513. It still crashes the same. No better, no worse. Most of our Access clients are still in the 19.6 range. We have a snapshot of the VM if we need to roll back.
This is the latest I was able to get from support regarding this issue:
"The version you are on is the latest version of Linux that is available, however the security issue that you received an email about does not affect Linux servers so you are able to bring your server back online with no issues. The team is working on putting out an update for Linux to bring it more in line with the rest of the product but the port to Mono that we have been using has been having issues. Our CTO talks about it in this forum posthttps://control.product.connectwise.co..."
How about that..
the security issue that you received an email about does not affect Linux servers
It's a really strange phenomena, its almost like this things keep happening over and over on Windows but not as frequently on Linux. There are several reasons for this and it's also the reason i'm not migrating over to a Windows server. Not only do you have to manage and buy licenses for this and that, you also need to keep on your toes for the next "bug/s" Microsoft releases. Aside from the server market, If Microsoft were serious, then Windows 10 Pro would be like LTSC or let smaller companies buy LTSC licenses without the stupid 5 licenses cap. /yet another MS rant
Windows is still based upon windows nt which is more than 25 years old...more and more crap gets piled on top...There is zero chance of windows being secured as the latest severe vulnerability is more than 10 years old(the dns server bug)...every time Microsoft says NT has been rewritten just look at the security vulnerabilities and you see things like the dns flaw that goes back years or decades. With the official position being linux is being abandoned after the next release from what i saw i am curious what folks linux migration plans are.
I've just spoken to support as no one seems to be replying to comments in here from CW.
You sent us an email stating you had a critical security flaw. We like many, have not been able to use your product for the best part of 7 months. We are on version 20.1.27036.7360. I cannot see any patch download. Please confirm.
Their response was that the version you need is ScreenConnect_20.1.29489.7513_Release.tar.gz Stable 7/27/2020 76 MB Linux.
This is, therefore, an UPGRADE and we must Pay for it. I asked exactly what the issue was and was given the blurb "To limit the potential for exposure of this vulnerability, we will be holding our normal Security Bulletin update to the Trust site until Wednesday, August 5. No information has been given currently https://www.connectwise.com/company/trust"
We've held off paying again for the support this year because of the issues with Linux. They've stated that it can be installed on a Windows 7 or 10 box with .net 4.7.1 rather than a server OS to avoid licencing costs.
I would appreciate your opinions please on what to do next?
We don't ever distribute 'patches', just new builds. If your license supports 20.1, it will support the new build of 20.1 without having to pay anything. We don't intend to charge you. Have you tried to install ScreenConnect_20.1.29489.7513_Release.tar.gz?
Speaking of new builds, is there any update on the 20.2 beta that was closed testing? I am eager to know if these mono issues have been cleared away. SC was one of my easiest updates at the time.
I haven't tried no because the support agent said I had to pay to upgrade. I don't want to break this installation any further than it already is.
I had the same thought process in "breaking the system further", but as yet my system has been the same, just running the latest version available.
@Jake Morgan is right too. Although I did find it interesting that my system used to tell me I was eligible to 20.4 (i think) and now it says that I'm only eligible to 20.1. I presume that is because there is no mono build?
Of course make sure you turn off automatic client upgrades and then make a backup of your system before you do upgrade, this way if it goes south you can just revert the CWC server and don't have to uninstall all the clients.
I decided to migrate from CentOS to Ubuntu for various reasons and upgrade from 20.1.27x to 20.1.29xThe documentation is a bit all over the place and I probably made a mistake by starting with Ubuntu 20.04LTS rather than 18 but here is a process that works in case it helps anyone-https://servicemax.com.au/news/connectwise-control-v20-1-on-ubuntu-20-04/
Since it seems at least a few of you on here aren't getting emails from connectwise, here is the latest missive.
I got this one - both on my personal and company accounts.
Tried to see what information I could on the security issue but had issues trying to look into it on my phone
Yeah I didn't get it. Thanks for sharing.
@Jake Morgan - Wouldn't it be possible to have this added to CWC in the admin portal as announcements for security? Then at least it isn't public, but accessible too.
I understand the need to keep things under wraps, but only an admin would be able to see it. Just a thought.
As of yesterday, I've started moving my agents from my Linux server to a new Windows server running in Azure. I'm so amazed how much faster the CW Control on Windows server is. Switching between access groups is very snappy. Reloading the Admin page is soooo much faster. Only difference between the two hardware wise is 4GB more of RAM on the Windows server (8GB total). Same vCPU cores (2). I may move over my Ubiquiti UniFi over to the Windows server if it doesn't slow things down.
My linux install WAS that responsive BEFORE 19.x. SSD's are your key here
actually i have over 125 agents on ubuntu Linux right now on a server with hard drives. ssd's aren't the issue..RAM is. The machine with SC on it has 16 gigs of ram and the SC process is nearing 6 gigs of usage. I also have 8 cpu's for it as well..although it rarely goes above using 4 i have seen it chew up all 8 when initializing. If you are having to run ssd's to speed up your program..most likely the issue is cpu and/or RAM as if you do not have enough ram the machine will swap like mad..which then hammers your HDD's. If you are swapping that much you will only wear out your ssd's faster as you burn up write cycles with swap file usage.
Was the Linux server also in Azure? .. I would be shocked if Windows make that type of difference alone, by default Linux will be faster unless you really kill the system setups/configuration (yes mono doesn't help but wouldn't be as large of an issue).
Our Linux Control install is still very responsive when it comes to changing web menus and such (until mono freaks and kills the memory or CPU for a few minuets at a time)
No the Linux server is on Linode, the $20/month plan, 2 vCPU and 4GB RAM. 99% sure it is on an SSD based host server.
SC should be fast on both Linux and Windows ... for small installations. It only slows as your installation grows and places more demands on the system.
Under Windows the slowdown is graceful and only occurs after fully and efficiently maxing CPU resources. Performance will generally improve proportional to CPU resources added.
Under Mono things go sideways usually well before CPU resources are exhausted. Adding more CPU resources can help, but not predictably. We don't support performance issues under Mono-- usually customers are elated to find they can cut their CPU resources by 80% and get better performance by moving to Windows. *HINT* *HINT*
Suggesting that we move our installs to Windows is just not cool, especially after you just had an exploit that apparently only effected Windows server installs.
Or are the linux servers affected as well?
How about the client installs?
Nobody ever answered the question of whether client installs are vulnerable.
@JakeMorgan Why the silence on the issue of the .NET upgrade? I'm not understanding why there seems to be no willingness to even have a conversation on the topic. Even if Linux is out of the picture .NET 4.x is a legacy platform at this point. Is CC going to remain on .NET 4 indefinitely? That doesn't bode well for Windows users either.
You speak of the financial resources being used for Mono maintenance, but is the budget for CC overall (including your primary Windows target) so small that a basic upgrade of the core platform is not on the table? And if it is on the table, why wait a year or two when you could resolve all this mess now?
Can we at least have some discussion on the issue from people on the development team? I just don't get this mentality that if we ditch Linux everything is going to be perfect from now on...
I just don't have an answer I think y'all will like very much. Issues:
- Although 95% of code would be preserved, the code that touches certain specific technologies (wcf, asp.net, appdomains) would have to be reworked, and that would take a good while
- We depend on extensions ourselves, and I don't think it's an acceptable solution to not support them on a platform
- We're very happy with .NET 4.8 on Windows. We use the latest C# features and haven't needed a library yet that isn't supported. It doesn't feel old, yet.
- We wouldn't move to .NET core without evaluating many other architectural opportunities to execute at the same time. Many of these opportunities could adversely affect on-prem in general
- Best case would be at least 6 months before anything would be ready with .NET core, and many people here haven't shown that kind of patience
Thanks for being forthcoming, I saw the writing on the wall awhile back and am working on moving to Windows.
Thanks for the response. I get that .NET 4.8 isn't "old" yet, but it clearly is a dead-end platform. I don't think it's fair to compare it to some of the previous .NET generations when Microsoft was still in their "support legacy forever" mentality. That just isn't the Microsoft of today which aggressively pushes mandatory updates and frequently iterates it's development platforms. I'm relatively sure .NET 4 code will still be able to run years down the road, but there will be no new code features from Microsoft to be taken advantage of and there will likely be a growing number of issues answered by Microsoft with something along the lines of "just upgrade your code to .NET 5+ and that problem goes away". There are also costs/challenges involved in waiting a long time to upgrade platforms, as it's a lot harder to make architectural changes if your upgrading to a platform 2 or 3 generations later than your existing platform.
Extensions that are super-important or used by CW would certainly need to be migrated, but are there really that many of them that are that important (and contain a lot of incompatible code)? Obviously I don't know your actual extension use but I'd guess this just isn't really a huge issue all things considered.
I'm not sure how to respond to the evaluation of "other architectural opportunities". If you're saying other potential changes would make it harder for CW to even offer an on-premise product, that would be an intentional direction change that goes against what has been promised both during the ScreenConnect acquisition and for years after. I know some companies do break their promises but I would hope that isn't a future outcome for CC--that would be horrible and you'd lose a lot of customers (a lot more than just Linux users). You'd also gain a bad reputation that would be hard to shake for a very, very long time.
In regards to a timeline for a .NET upgrade, I've suggested a few times that I'd be willing to wait a reasonable amount of time if there was a clear commitment and plan for the process. I know some people here (especially those who are experiencing major stability problems) aren't going to be very patient, but that doesn't mean there aren't Linux users that would be appreciative of a timeline and willing to wait. Just because some customers aren't willing to wait doesn't mean that the whole idea should be abandoned. I'm sure plenty of the customers who ended up migrating to Windows out of necessity would happily return to Linux if the issues were resolved. If the commitment was actually made and there was transparency throughout the process I think you'd see a lot more goodwill from people here. Having a "temporary" Linux release that is stable enough (such as is being worked on with the private beta) would also help during this time.
By extensions, I think Jake means those that are automatically installed when a cloud instance is deployed: Cloud Authentication; In-Application Help; Remote Workforce.
Also, features exclusive to Premium and Access packages on the cloud are provided via extensions.
Advanced Configuration Editor is one extension that's essential to cloud instances, since you can't manually edit the web.config file on hosted servers. Extension Developer is another.
@Simon Sure, but I find it hard to believe that:
1) those extensions are very complex
2) those extensions have a ton of incompatible code that would necessitate a huge amount of refactoring
I still don't see extensions as being a big deal in this context. Just figure out what extensions are absolutely necessary and make sure those work with a .NET Core based CC from the start. After that any other extensions that are worthwhile could be migrated over time (or just replaced with new extensions). If the core project code is mostly in a good place to make the transition to .NET Core I'd think a single developer could probably get a very rough alpha up and running within a couple weeks (enough to at least start figuring out what components/extensions can't run under .NET Core).
I have no issues with patience if you (you as in ConnectWise) are communicating whats going on and have some kind of portal (or send out mails) on status reports. I've mentioned this before and being this quiet for half a year is not gaining me (or anyone else) trust in ConnectWise as a company. What has been communicated in the last month should have been done way earlier but what is done is done.
I could go on with arguments back and forth but I feel that you have already given up on Linux and are just trying to fix the current version to slowly let it fade while pushing customers to a Windows setup or cloud instead as thats whats been communicated for quite a while from both customer emails and the forum.
One could hope that you come to your senses but what are the chances, what hope is really left other than migrating to Windows? Wouldn't surprise me if you in time will force customers over to your "cloud" solution and kick your on-prem customers on the balls as well.
Guess i'm not renewing anymore..
I agree with your sentiment on how they handled the Linux situation - the issue was too big for the amount of money they're seeing on renewals from the Linux on-prem customers and they should have simply said that upfront instead of stringing us along. I'll migrate to Windows and keep going until it comes to a "cloud" or nothing option.
Jake - Thanks again for being more transparent here. I will mention however that there was quite a bit of patience for the first 4-5 months of this situation. People got frustrated with the lack of progress and communication.
On a side note to everyone else, migrating to Windows will have some additional licensing costs above and beyond the actual Windows license. I verified with our MS rep that a CAL (or External Connector license) would be required for each named user (tech) whether they authenticate with AD or not.
CC runs on 10, could you do that to avoid the CAL requirement?
I love how Jake somehow blames the users of the software for the issues in the system. It is pretty funny. If Connectwise just kept everything up to date we wouldn't have this issue. Lazy and poor form.
Well I don't know what to say why SC crawls on my Linux VPS. Per support years ago, I run a weekly database routine of compacting, removing certain info, etc to try and keep it small. Database file right now is over 400MB on the Linux server (I haven't checked the Windows server). I have over 500 access agents, though only half are currently ever connected (i.e. turned on) at any one time. Now that I've moved over to Windows, it really is a night and day difference in dashboard performance. Searching for a device is snappy, as is the Admin page loading. I've even gone through the Linux SC server and removed more than half of the agents (since they are now redirected to the new Windows server) and the damn Linux SC server is still freaking slow. (Customers who haven't used PCs in awhile due to Covid19, so I'm leaving the Linux SC server in place while I migrate them over). I'm still having to restart the SC service daily on the Linux server since it locks up but not yet with the Windows server *fingers crossed*.
The "Alpha" Focus Group build is very fast. Until it stops responding. I honestly think that CW is starting/trying to figure it out. I'd probably hang in there for an official Linux release of the next version. Note: Not a ConnectWise/Screenconnect Employee.
No it was not, stand alone
I just got the email urging to upgrade/patch - the downloads page shows a 20.1 from 7/27, I'm running 20.2 (20.2.27296.7376) that I installed in January and has been more or less stable for me. Should I be going to this 20.1 from 20.2 to address this? And is there a writeup on what the vulnerability is?
The writing is on the wall to go to Windows, but outside of some mono memory leaks where I had to bump up memory while waiting on a new version, I've been able to work with a $5/mo DigitalOcean VPS (512 MB, then 1 GB) for most of the last 7 years.
I ma not going to bear the extra costs of a Windows server. Looks like i'll be trying some alternative self-hosted options.
It'll run on 10: https://docs.connectwise.com/ConnectWise_Control_Documentation/On-premises/Get_started_with_ConnectWise_Control_On-Premise/Server_system_requirements - my plan so far is to get two used ThinkCentre Tiny units and put it on there and do image backups to go back in time or restore to the backup computer if needed. That'll give me an acceptable hardware and software safety net.
I hate the entire idea of it along with the associated overhead vs a cheap Linux VPS with quick snapshots, but I'd rather stay current and I'm grandfathered in on inexpensive upgrades.
This is exactly what we are doing C2020
why wouldn’t you just put it on AWS Lightsail?
Learn something new everyday. Looks like Lightsail is their direct answer to DigitalOcean, with Windows availability and snapshots. Latest AWS I used was SES for sending from a PBX :)
Thanks for this. I'll try at the 512MB instance and likely go to 1GB
I just tried the 512mb instance and it is seriously slow with CWC 20.1.29489.7513 - pretty much unusable. And you can't add resources to the Lightsail instance - you have to start again. I will try the 1gb but the recommended resources are 4gb RAM and 2 core processor for 200 clients - which would be $40 per month
I have 200 clients and CWC works (or used to work) very well on Ubuntu 18.04 with 1gb RAM and 20gb storage for $8 per month
I would really like to stay with Ubuntu on Digital Ocean but performance is very patchy, I never know how long it's going to take before my clients become available, and there are SSL chain errors.
Now we hear that it will be another few weeks until the beta testing is complete...
did know windows would even run with 512mb if ram... seriously though, thought 2GB would have been the minimum. Lightsail still seems like a better option then doing some half baked Local PCs acting like servers idea (IMO).
Yes I used to run it on a laptop at home, but every time I went travelling abroad I feared a power cut would knock it out. Had it working on AWS with a Windows 2008 Server originally but after the free tier ended it was quite expensive to run. Running it on Digital Ocean has been great (until this year).
I've now set up CW on Lightsail with 1gb RAM and single core processor. Initial impressions suggest it might be OK.
CW on Lightsail with 1gb Ram Server 2016 and 40gb SSD for $12 per month. It is seriously fast and snappy. Very responsive. Quite a suprise I have taken the plunge and migrated my 245 clients. I set the paging file size at 4096mb as a backstop in case it exhausts the RAM, but most I ever have running is 3 concurrent sessions. Also the SSL chain is working properly with no errors.
When you migrated did you lose any clients that were inactive?
Shouldn't matter if you're using a DNS name rather than just an IP
Going by what their rep said: "01:36 PM - asingh: However in case the agents which are offline during this migration they might need to be reinstalled manually"
As long as you are using DNS/domain name, when the offline agents come back online, they will check in as normal. Ive moved to windows, I had no issues, all clients came back as they should, Even the ones that were powered down for the weekend.
k thank you
If by "inactive" you mean clients that were offline - no, they all migrated - online and offline. I haven't lost anything.
After 1 week running CWControl on Lightsail with 245 clients, my experience is positive. CWC is quick, responsive and stable, and on a system with only a single core processor and 1gb RAM. It is working very well, and I am very relieved. I have now upgraded to version 20.8.29679.7530. Lightsail does not offer automatic snapshots of the server installation, whereas this was available on Digital Ocean / Ubuntu. But I make daily and weekly backups to Dropbox of the ScreenConnect installation folder which is the important part. Good luck to everyone holding out for the Linux version to improve.
Can we please get an update on what is happening? Beta guys? CWC???
Back when this fiasco started 20.2 became available for a brief window. I have been running 20.2.27450.7387 for a wile now. I reported a problem i am having with the system not letting go of connections once they are made. sometimes i have to restart the entire SC process to get the stupid thing to release. I posted a report on this very forum and attached the requested information. Is this bug(which i can find the link to) part of the dev efforts?
Got a ConnectWise Newsletter with the subject "Not Operating at 80% Utilization? You’re Losing Money." today. Thank god SC is running most of the time at 100%, no money to lose on my end ;-)
Received a connectwise survey today asking how the FocusGroup build is... not sure why I would have gotten a survey like that when there are active items open in the private support thread. I have one engineer that reached out. Seems like they might be setting up a 1-on-1 session to review my server install and help debug things. Perhaps this will shed light on some of the linux unresponsiveness. Try to keep you all updated here as well.
Thanks mate. Appreciate the updates.
We now migrated from Linux to Windows. As we didn´t have much trust in the Windows firewall on a system open to the world, we installed an OPNSense firewall in front of the CWC Windows server. Now we have to administer 2 servers instead of only one small linux machine.
Migration was a charm though, installed the same version as on linux, copied some files, ran repair installation and copied some settings from the old web.config file. Ran some tests with guests where we changed the IP for the DNS name in their local hosts file. After those checks we changed the IP in the DNS server (it helps if you lower the TTL for your A/AAAA records some days before to i.e. 5 minutes). All machines were able to connect. Feels a lot snappier while controlling a guest.
We would love to see the linux version in the future again. I think by dumping Mono it should be able to be on par with the Windows version, but for the moment it was more secure for us to be able to use the latest version. Would be great to reduce the administration effort to only one linux box again in the near future.
do you think we will be able to migrate from windows back to linux? (If for some reason they decide to properly support linux again..)
Yes, it should be fairly easy. Most important is that versions do match.
Here is a good documentation on how to do this:
I get the need for the feeling to move off --- I am starting to have more confidence in the Focus Build. Hopefully CW will pre-release it for people.
@caitlyn any update for this week? Hopefully there is good news :)
I don't see an entry for August 5 on the Security Bulletin page. Whatever happened to that?
So I've succumbed. 7 months and I couldn't hack it any more. Windows VPS server, CWC 20.9 installed. It does seem smooth, but I haven't got any Unattended setup yet.
Is anyone able to share their setup using IIS reverse proxy? I am using LetsEncrypt to generate a certificate (works brilliantly) but then I need IIS to apply it. Issue with that is CWC then needs a reverse proxy (this was all done with Linux easily enough and Nginx).
Mine can display, but something is preventing access to a file and throws an error every now and again: Failed to load resource: 502 - https://FQDN/Services/PageService.ashx/GetHostSessionInfo
I haven't tried the reverse proxy on my Windows Server yet. Was hoping to use straight up SSL implementation within Control; hopefully they've finally updated it to remove insecure/outdated SSL protocols (hence why I had Apache reverseproxy on Linux because of that).
I used the straight up SSL implementation with Control, using the SSL configurator. I have a 2 year SSL cert, rather than using Let'sEncrypt. It worked like a charm with no SSL errors
Have you tested the SSL implementation using https://www.ssllabs.com/ssltest/ ?
I just tested it with this basic checker https://www.sslshopper.com/ssl-checker.html#hostname=support.calmit.orgOn this thorough it test scores a B due to use of older protocols, it seemshttps://www.ssllabs.com/ssltest/analyze.html?d=support.calmit.org
The TLS ciphers are defined at the Windows level in the registry. There is no way for us to override for our application. We don't mess with system-wide settings in the registry-- that's up to you. I'd imagine we have a better article on this, but I know here is where Microsoft describes it:
Yep this is very important.
As always have a BACKUP!!! Before messing around.
I've removed the TLS v1.0 and v1.1 using this website - https://improveandrepeat.com/2020/03/how-to-disable-tls-1-0-1-1-and-ssl-on-your-windows-server/
Used https://www.ssllabs.com/ssltest/ initially but it was caching the test so I used this instead - https://www.digicert.com/help/ (found out later that you clear the cache easily enough whoops)
Also removed an insecure ciphers - https://docs.microsoft.com/en-us/powershell/module/tls/disable-tlsciphersuite?view=win10-ps
I went through using https://www.ssllabs.com/ssltest/ again, and removed all the insecure ciphers.
Be interested to know how others rank using the inbuilt webserver of CWC.
IISCrypto works really well for setting protocols/ciphers without having to manually edit the registry keys, and can help create/restore backups of the keys. Disclaimer, it's not our tool so use at your own risk :) but it's available for free:https://www.nartac.com/Products/IISCrypto/
Thanks for that - I removed TLS v1.0 and v1.1 and the insecure ciphers as you did, and also now scoring "A" on https://www.ssllabs.com/ssltest/
SSL certificate configurator made it easy to setup. Purchased a cheap paid SSL certificate for all of $8.00 for a year. Ran configurator to generate CSR. Copy/pasted CSR into vendor's website. Received new cert. Uploaded cert onto the Windows Server desktop. Configurator found the cert and installed.Out of the box CW SSL implementation, I received a Score of "B" due to TLS 1.0 and 1.1 being enabled on the Windows Server that I am using (also default config). Once I disabled TLS 1.0 and 1.1 via reg key in the link above, the Score is now "A".
In several years, i've been using https://certifytheweb.com/ for those Windows servers running software which can't run on Linux (if that could be of any help for you since it helps with some of the configuration)
Yes I use that as well. Works brilliantly easy enough. But do you run IIS as a reverse proxy?
I'm really keen to hear from anyone who runs it and what their settings are in IIS.
As we´re not so convinced of a Windows machine being directly connected to the internet we installed an OPNSense firewall in front of it and have HAProxy doing the SSL offloading. The control channel for CWC is the only thing that is port forwarded (DNAT).
No, not currently. I've instead minimized the possible attack surface by only allowing certain CIDR's to connect but i'm planning to use a 5$ vps with nginx to act as a reverse proxy while still limiting IP's.
I do remember there was some hassle to setup the IIS on Windows compared to Debian/Ubuntu+Nginx with various illogical issues.
Did you try the search term "Failed to load resource: 502 IIS" in your favourite search engine?
Using a OPNSense firewall (or other hardware firewalls) in front of a Windows server as Chris mentioned is otherwise a wise choice if your setup currently is wide open albeit with some extra overhead, which is why I again prefer to run linux servers and not have to babysit Windows.
Keep in mind for those of you running (or thinking about) CWC on a regular Windows Professional setup that you have several services and preinstalled software running in the background which is totally pointless and impacts security and stability. Windows server and the LTSC editions are both better in this regard and doesn't have preinstalled apps for Xbox, Cortana, Spotify, Office, Games etc and also has less risk for certain feature updates (old as new) to impact security even further since it's only security updates thats coming through.
So unless you have strict firewall rules for both incoming and outgoing, Windows itself will open up various ports for various dumb shit which can be used against your server. Maybe not always a huge risk but it's still a risk to account for with all these automated bots crawling the web.
Customer support service by UserEcho