- Bug Reports
-
Bugs
Mono eating too much memory (Linux)
Hello,
I am running ScreenConnect 6.2.12963.6312 on CentOS (Linux) and as of late I keep getting "out of memory" errors with Mono via /var/log/messages. I have not done any major changes on my server, and keep it up to date regularly. Things have been running fine for years until perhaps the last upgrade to ScreenConnect.
My server is a dedicated server - it has a Xeon processor 1270v3 (3.5 GHz quad core with hyperthreading), running a virtual machine instance of CentOS release 6.9 (Final). The virtual machine has 4GB of memory.
Here is a snippet from /var/log/messages:
Jun 7 19:22:12 xeon3 kernel: lowmem_reserve[]: 0 0 0 0
Jun 7 19:22:12 xeon3 kernel: DMA: 5*4kB 4*8kB 2*16kB 2*32kB 1*64kB 4*128kB 2*256kB 0*512kB 2*1024kB 2*2048kB 0*4096kB = 7380kB
Jun 7 19:22:12 xeon3 kernel: Normal: 1019*4kB 699*8kB 322*16kB 181*32kB 93*64kB 332*128kB 121*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 105668kB
Jun 7 19:22:12 xeon3 kernel: HighMem: 68*4kB 8*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 336kB
Jun 7 19:22:12 xeon3 kernel: 11236 total pagecache pages
Jun 7 19:22:12 xeon3 kernel: 11096 pages in swap cache
Jun 7 19:22:12 xeon3 kernel: Swap cache stats: add 1678912, delete 1667816, find 4150814/4208192
Jun 7 19:22:12 xeon3 kernel: Free swap = 0kB
Jun 7 19:22:12 xeon3 kernel: Total swap = 835580kB
Jun 7 19:22:12 xeon3 kernel: 1048560 pages RAM
Jun 7 19:22:12 xeon3 kernel: 821762 pages HighMem
Jun 7 19:22:12 xeon3 kernel: 43834 pages reserved
Jun 7 19:22:12 xeon3 kernel: 9549 pages shared
Jun 7 19:22:12 xeon3 kernel: 974561 pages non-shared
Jun 7 19:22:12 xeon3 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jun 7 19:22:12 xeon3 kernel: [ 570] 0 570 663 1 3 -17 -1000 udevd
Jun 7 19:22:12 xeon3 kernel: [ 939] 0 939 661 1 2 -17 -1000 udevd
Jun 7 19:22:12 xeon3 kernel: [ 942] 0 942 661 1 1 -17 -1000 udevd
Jun 7 19:22:12 xeon3 kernel: [ 1310] 0 1310 3254 23 2 -17 -1000 auditd
Jun 7 19:22:12 xeon3 kernel: [ 1344] 0 1344 9553 567 1 0 0 rsyslogd
Jun 7 19:22:12 xeon3 kernel: [ 1406] 25 1406 28558 4882 0 0 0 named-sdb
Jun 7 19:22:12 xeon3 kernel: [ 1444] 99 1444 2915 199 1 0 0 openvpn
Jun 7 19:22:12 xeon3 kernel: [ 1489] 495 1489 20755 849 2 0 0 opendkim
Jun 7 19:22:12 xeon3 kernel: [ 2886] 0 2886 2168 3 2 -17 -1000 sshd
Jun 7 19:22:12 xeon3 kernel: [ 2897] 38 2897 1407 36 1 0 0 ntpd
Jun 7 19:22:12 xeon3 kernel: [ 2936] 0 2936 1345 1 0 0 0 mysqld_safe
Jun 7 19:22:12 xeon3 kernel: [ 3395] 27 3395 262755 143148 2 0 0 mysqld
Jun 7 19:22:12 xeon3 kernel: [ 3436] 0 3436 720 29 3 0 0 dovecot
Jun 7 19:22:12 xeon3 kernel: [ 3437] 97 3437 685 23 1 0 0 anvil
Jun 7 19:22:12 xeon3 kernel: [ 3439] 0 3439 684 32 1 0 0 log
Jun 7 19:22:12 xeon3 kernel: [ 3452] 0 3452 2232 1 0 0 0 saslauthd
Jun 7 19:22:12 xeon3 kernel: [ 3453] 0 3453 2232 1 1 0 0 saslauthd
Jun 7 19:22:12 xeon3 kernel: [ 3454] 0 3454 2232 1 1 0 0 saslauthd
Jun 7 19:22:12 xeon3 kernel: [ 3455] 0 3455 2232 1 3 0 0 saslauthd
Jun 7 19:22:12 xeon3 kernel: [ 3456] 0 3456 2232 1 2 0 0 saslauthd
Jun 7 19:22:12 xeon3 kernel: [ 3467] 494 3467 3787 547 2 0 0 postgrey
Jun 7 19:22:12 xeon3 kernel: [ 3546] 0 3546 3327 26 0 0 0 master
Jun 7 19:22:12 xeon3 kernel: [ 3558] 89 3558 3596 105 1 0 0 qmgr
Jun 7 19:22:12 xeon3 kernel: [ 3576] 0 3576 1476 22 0 0 0 crond
Jun 7 19:22:12 xeon3 kernel: [ 3600] 0 3600 81829 2060 0 0 0 fail2ban-server
Jun 7 19:22:12 xeon3 kernel: [ 3691] 0 3691 502 1 0 0 0 mingetty
Jun 7 19:22:12 xeon3 kernel: [ 3693] 0 3693 502 1 3 0 0 mingetty
Jun 7 19:22:12 xeon3 kernel: [ 3698] 0 3698 502 1 1 0 0 mingetty
Jun 7 19:22:12 xeon3 kernel: [ 3700] 0 3700 502 1 3 0 0 mingetty
Jun 7 19:22:12 xeon3 kernel: [ 3702] 0 3702 502 1 3 0 0 mingetty
Jun 7 19:22:12 xeon3 kernel: [ 3704] 0 3704 502 1 1 0 0 mingetty
Jun 7 19:22:12 xeon3 kernel: [ 4672] 89 4672 3642 36 0 0 0 tlsmgr
Jun 7 19:22:12 xeon3 kernel: [29682] 0 29682 1230 211 1 0 0 config
Jun 7 19:22:12 xeon3 kernel: [13051] 0 13051 763 3 1 0 0 screenconnect
Jun 7 19:22:12 xeon3 kernel: [16817] 498 16817 1764 42 3 0 0 imap-login
Jun 7 19:22:12 xeon3 kernel: [16818] 498 16818 1764 43 3 0 0 imap-login
Jun 7 19:22:12 xeon3 kernel: [16825] 5000 16825 1059 128 1 0 0 imap
Jun 7 19:22:12 xeon3 kernel: [16826] 5000 16826 1043 48 2 0 0 imap
Jun 7 19:22:12 xeon3 kernel: [12269] 0 12269 18804 480 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19922] 0 19922 3047 3 0 0 0 sshd
Jun 7 19:22:12 xeon3 kernel: [19928] 0 19928 1378 3 1 0 0 bash
Jun 7 19:22:12 xeon3 kernel: [ 6403] 0 6107 767739 577098 1 0 0 mono
Jun 7 19:22:12 xeon3 kernel: [17953] 97 17953 3405 249 1 0 0 auth
Jun 7 19:22:12 xeon3 kernel: [18359] 0 18359 3405 260 3 0 0 auth
Jun 7 19:22:12 xeon3 kernel: [18362] 5000 18362 1023 115 3 0 0 imap
Jun 7 19:22:12 xeon3 kernel: [18610] 48 18610 33171 14047 2 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [18927] 48 18927 33147 14946 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [18998] 48 18998 29059 9753 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19002] 48 19002 32900 12509 2 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19138] 89 19138 3569 164 3 0 0 pickup
Jun 7 19:22:12 xeon3 kernel: [19181] 48 19181 33155 12180 1 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19252] 48 19252 33155 14846 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19305] 48 19305 32899 13820 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19393] 48 19393 28795 10018 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19394] 48 19394 33658 13610 1 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19395] 48 19395 28795 10309 3 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19488] 48 19488 33147 13654 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19491] 48 19491 28795 9598 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19492] 48 19492 29051 10418 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19516] 48 19516 29051 9323 1 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19557] 48 19557 28283 9483 1 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19558] 48 19558 29051 10229 1 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19678] 0 19678 3405 261 3 0 0 auth
Jun 7 19:22:12 xeon3 kernel: [19689] 0 19689 3405 261 2 0 0 auth
Jun 7 19:22:12 xeon3 kernel: [19793] 48 19793 28027 8690 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19812] 48 19812 28283 8522 1 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19850] 48 19850 28027 8388 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: [19880] 89 19880 3396 174 1 0 0 smtp
Jun 7 19:22:12 xeon3 kernel: [19881] 89 19881 3396 175 2 0 0 smtp
Jun 7 19:22:12 xeon3 kernel: [19883] 89 19883 3356 164 1 0 0 bounce
Jun 7 19:22:12 xeon3 kernel: [19901] 48 19901 27515 8268 0 0 0 httpd
Jun 7 19:22:12 xeon3 kernel: Out of memory: Kill process 6403 (mono) score 580 or sacrifice child
Jun 7 19:22:12 xeon3 kernel: Killed process 6403, UID 0, (mono) total-vm:3070956kB, anon-rss:2308240kB, file-rss:152kB
My "top" command, sorted by memory is below. Note that the server isn't anywhere near crazy busy with apache or other processes. That said, you can see mono is eating 1.9GB only 10 minutes of restarting the screenconnect services. This is insane, considering I'm not connected to any sessions. I have a grand total of 3 active "access" clients, and 7 "support" clients which have not even connected to their sessions yet.
top - 12:36:41 up 19 days, 22:21, 2 users, load average: 1.48, 1.27, 1.18
Tasks: 231 total, 2 running, 229 sleeping, 0 stopped, 0 zombie
Cpu(s): 15.0%us, 11.3%sy, 0.0%ni, 67.2%id, 3.9%wa, 0.8%hi, 1.8%si, 0.0%st
Mem: 4019160k total, 3906876k used, 112284k free, 137216k buffers
Swap: 835580k total, 416164k used, 419416k free, 163228k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25662 root 20 0 2189m 1.9g 3156 S 0.0 49.1 0:49.64 mono /opt/ScreenConnect/Bin/ScreenConnect.Service.exe startservices 7 21424 10
3395 mysql 20 0 1015m 522m 3804 S 12.6 13.3 1754:49 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/li
29263 apache 22 2 130m 62m 5852 S 0.0 1.6 0:57.00 /usr/sbin/httpd
4714 apache 20 0 128m 58m 5856 S 0.0 1.5 0:44.16 /usr/sbin/httpd
25616 apache 20 0 129m 58m 5684 S 0.0 1.5 0:14.33 /usr/sbin/httpd
18557 apache 20 0 129m 57m 5848 S 0.0 1.5 1:05.69 /usr/sbin/httpd
20237 apache 20 0 129m 57m 5884 S 0.0 1.5 1:02.57 /usr/sbin/httpd
15004 apache 20 0 129m 56m 5760 S 17.6 1.4 0:32.22 /usr/sbin/httpd
3042 apache 20 0 128m 54m 5860 S 0.0 1.4 1:20.46 /usr/sbin/httpd
24587 apache 20 0 128m 53m 5756 S 0.0 1.4 0:12.83 /usr/sbin/httpd
14442 apache 20 0 112m 42m 5816 S 0.0 1.1 0:34.36 /usr/sbin/httpd
30441 apache 20 0 112m 41m 6436 S 16.3 1.1 0:10.23 /usr/sbin/httpd
30458 apache 20 0 109m 41m 6396 S 0.0 1.1 0:06.70 /usr/sbin/httpd
30965 apache 20 0 109m 41m 6408 S 0.0 1.1 0:08.46 /usr/sbin/httpd
29685 apache 20 0 111m 41m 6324 S 0.0 1.1 0:08.60 /usr/sbin/httpd
30948 apache 20 0 109m 41m 6416 S 0.0 1.0 0:06.39 /usr/sbin/httpd
29531 apache 20 0 112m 40m 6052 S 0.0 1.0 0:04.66 /usr/sbin/httpd
30966 apache 20 0 110m 39m 6440 S 0.0 1.0 0:05.38 /usr/sbin/httpd
565 apache 20 0 109m 39m 6332 S 19.0 1.0 0:02.64 /usr/sbin/httpd
31148 apache 20 0 109m 38m 6424 S 0.0 1.0 0:07.93 /usr/sbin/httpd
27232 apache 20 0 109m 38m 5720 S 0.0 1.0 0:14.49 /usr/sbin/httpd
29548 apache 20 0 109m 38m 5988 S 0.0 1.0 0:10.78 /usr/sbin/httpd
1118 apache 20 0 109m 38m 6336 S 0.0 1.0 0:02.37 /usr/sbin/httpd
26820 apache 20 0 109m 38m 5768 S 0.0 1.0 0:15.87 /usr/sbin/httpd
1543 apache 20 0 97892 28m 5232 S 0.0 0.7 0:00.70 /usr/sbin/httpd
1406 named 20 0 111m 20m 1696 S 1.3 0.5 144:21.18 /usr/sbin/named-sdb -u named -4 -t /var/named/chroot
3600 root 29 9 319m 10m 1580 S 33.2 0.3 1629:46 /usr/bin/python /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /
2326 root 20 0 47296 6164 1592 S 11.0 0.2 12:02.03 /usr/bin/php /var/www/vhosts/infopackets.com/httpdocs/lists/admin/index.php -c/v
664 root 20 0 13620 3960 2932 S 0.0 0.1 0:00.01 dovecot/auth -w
1489 opendkim 20 0 83020 3512 1040 S 2.0 0.1 71:57.05 /usr/sbin/opendkim -x /etc/opendkim.conf -P /var/run/opendkim/opendkim.pid
8598 root 20 0 13620 3280 2248 S 0.0 0.1 0:00.08 dovecot/auth -w
Suggestions?
- Dennis
Customer support service by UserEcho
Wow, an entire month has gone by and still no response.
Since originally posting this issue, I completely uninstalled and reinstalled the product clean with a brand new configuration file and sessions database, and made only minimal changes to the configuration file. I've documented what I did here:
https://www.infopackets.com/news/10125/how-fix-uninstall-reinstall-screenconnect-server-mono-out-memory-error
Mono seemed to be behaving itself up until a few days ago, but now I'm right back where I was - in fact, the problem seems worse. Every click I make on a session eats 1 megabyte of memory. If I review a chat, another 300 megabytes gets eaten. If I review more than 3 chats I get "unknown error" in my browser and mono crashes and restarts on the server.
This is absolutely ridiculous. I've upgraded to the latest edition (ScreenConnect_6.3.13446.6374) and the problem has not been fixed!
I've sent an email to ScreenConnect sales and posted here, and have ZERO help from anyone. My license is going to expire on August 9th, 2017 and it seems I have little recourse but to keep reinstalling the product to "fix" the issue for a unknown duration of time until it creeps up again.
What can be done about this problem? How about connecting to my server and looking at it?
Good morning,
It's not clear, based on the provided information, why the mono process is consuming all available memory.
Is it possible that the mono process on your server is running as a 32 bit process?
What is the output of running the following command as root on your server (insert the mono PID where indicated):
If the process is running as 64 bit, the output of the aforementioned command should be something like:
Regards,
Ben
Hello,
Thank you for looking into this. Yes, mono is 32-bit.
[root@xeon3 vmware:/var/log]# LD_SHOW_AUXV=1 ldd /proc/12596/exe | grep AT_PLATFORM | tail -1
AT_PLATFORM: i686
According to this page, 32-bit Linux is supported for ScreenConnect server roles:
https://docs.connectwise.com/ConnectWise_Control_Documentation/Get_started/System_requirements#Linux
Thank you for confirming you're using a 32-bit implementation. I've so far been unable to reproduce the OOM behavior using a 32-bit install of CentOS 6.9 and Control 6.3.13466.
Please update this thread with the following info:
1) What is the size of your session database, located at App_Data\Session.db
2) How many access sessions are calling back to your server?
3) On average, how many concurrent active sessions (host and guest connected) does your server handle?
4) Have you made any customizations to your server (e.g.have you set Control to run behind nginx)?
5) What extensions have you installed?
Regards,
Ben
1) What is the size of your session database, located at App_Data\Session.db
-rw-r--r-- 1 root root 26M Jul 11 01:05 /opt/ScreenConnect/App_Data/Session.db
2) How many access sessions are calling back to your server?
Currently 4 Support sessions and 2 Access sessions.
3) On average, how many concurrent active sessions (host and guest connected) does your server handle?
Usually I only connect to 1 user at a time but sometimes I use 2 concurrent sessions as that is the maximum that my license can handle.
4) Have you made any customizations to your server (e.g.have you set Control to run behind nginx)?
I run: Drupal, Apache, Dovecot, PhpMyAdmin, ScreenConnect, Perl, PHP, Mysql, Fail2ban on the server primarily. I don't use nginix.
5) What extensions have you installed?
None.
Note the screen captures below which show a whopping 300mb RAM jump in only 1 minute time. All I did was click on 3 conversations from clients and clicked on their sessions.
You are more than welcome to connect to my server to review the issues if you cannot replicate it.
Hello,
As I mentioned previously: after running out of memory repeatedly (about a month ago), I've reinstalled ScreenConnect - but it wasn't until weeks later I've come into this problem once again. If you cannot reproduce the issues on your own virtual machine, then how about SSH'ing into my system to have a look around? This does not seem to be an unreasonable request.
I have had this memory leak issue for at least 4-6 months now. Upgrading to CentOS 64bit to fix this problem doesn't seem like a legitimate answer since (a) that would require reconfiguring my entire server which took weeks to set up; (b) your support page says Linux 32 bit is supported, and (c) there is no guarantee the issue will be fixed since you don't really know what the bug is in the first place.
- Dennis
Dennis,
Did you ever find the solution to this issue? I am experiencing the exact same thing on my server as well. I am running a 2GB VPS over at OVH and I have noticed this issue happening for a while. It gets to the point where I cannot even load SC.
No I did not find a solution to this. I was more or less told that the 32-bit version of Mono (part of screenconnect) was unpredictable and that upgrading to 64-bit OS was recommended. That said I have not had an 'out of memory' issue for quite a while now. You can try and reinstall screenconnect as that might help. Also, from what I have read, screenconnect seems to be fairly stable in a Windows environment if you have access to that. I have not tried, so I can't say.
I am running the 64 bit version and having the same problem...
I was debating about launching a Windows VPS over at OVH and installing it. I just have to deal with the issue of the moving 70+ computers.