-1
Waiting for information

Mono eating too much memory (Linux)

wubbaducki 3 years ago updated by Mkvandals 3 years ago 9

Hello,


I am running ScreenConnect 6.2.12963.6312 on CentOS (Linux) and as of late I keep getting "out of memory" errors with Mono via /var/log/messages. I have not done any major changes on my server, and keep it up to date regularly. Things have been running fine for years until perhaps the last upgrade to ScreenConnect.


My server is a dedicated server - it has a Xeon processor 1270v3 (3.5 GHz quad core with hyperthreading), running a virtual machine instance of CentOS release 6.9 (Final). The virtual machine has 4GB of memory.


Here is a snippet from /var/log/messages:


Jun  7 19:22:12 xeon3 kernel: lowmem_reserve[]: 0 0 0 0
Jun  7 19:22:12 xeon3 kernel: DMA: 5*4kB 4*8kB 2*16kB 2*32kB 1*64kB 4*128kB 2*256kB 0*512kB 2*1024kB 2*2048kB 0*4096kB = 7380kB
Jun  7 19:22:12 xeon3 kernel: Normal: 1019*4kB 699*8kB 322*16kB 181*32kB 93*64kB 332*128kB 121*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 105668kB
Jun  7 19:22:12 xeon3 kernel: HighMem: 68*4kB 8*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 336kB
Jun  7 19:22:12 xeon3 kernel: 11236 total pagecache pages
Jun  7 19:22:12 xeon3 kernel: 11096 pages in swap cache
Jun  7 19:22:12 xeon3 kernel: Swap cache stats: add 1678912, delete 1667816, find 4150814/4208192
Jun  7 19:22:12 xeon3 kernel: Free swap  = 0kB
Jun  7 19:22:12 xeon3 kernel: Total swap = 835580kB
Jun  7 19:22:12 xeon3 kernel: 1048560 pages RAM
Jun  7 19:22:12 xeon3 kernel: 821762 pages HighMem
Jun  7 19:22:12 xeon3 kernel: 43834 pages reserved
Jun  7 19:22:12 xeon3 kernel: 9549 pages shared
Jun  7 19:22:12 xeon3 kernel: 974561 pages non-shared
Jun  7 19:22:12 xeon3 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Jun  7 19:22:12 xeon3 kernel: [  570]     0   570      663        1   3     -17         -1000 udevd
Jun  7 19:22:12 xeon3 kernel: [  939]     0   939      661        1   2     -17         -1000 udevd
Jun  7 19:22:12 xeon3 kernel: [  942]     0   942      661        1   1     -17         -1000 udevd
Jun  7 19:22:12 xeon3 kernel: [ 1310]     0  1310     3254       23   2     -17         -1000 auditd
Jun  7 19:22:12 xeon3 kernel: [ 1344]     0  1344     9553      567   1       0             0 rsyslogd
Jun  7 19:22:12 xeon3 kernel: [ 1406]    25  1406    28558     4882   0       0             0 named-sdb
Jun  7 19:22:12 xeon3 kernel: [ 1444]    99  1444     2915      199   1       0             0 openvpn
Jun  7 19:22:12 xeon3 kernel: [ 1489]   495  1489    20755      849   2       0             0 opendkim
Jun  7 19:22:12 xeon3 kernel: [ 2886]     0  2886     2168        3   2     -17         -1000 sshd
Jun  7 19:22:12 xeon3 kernel: [ 2897]    38  2897     1407       36   1       0             0 ntpd
Jun  7 19:22:12 xeon3 kernel: [ 2936]     0  2936     1345        1   0       0             0 mysqld_safe
Jun  7 19:22:12 xeon3 kernel: [ 3395]    27  3395   262755   143148   2       0             0 mysqld
Jun  7 19:22:12 xeon3 kernel: [ 3436]     0  3436      720       29   3       0             0 dovecot
Jun  7 19:22:12 xeon3 kernel: [ 3437]    97  3437      685       23   1       0             0 anvil
Jun  7 19:22:12 xeon3 kernel: [ 3439]     0  3439      684       32   1       0             0 log
Jun  7 19:22:12 xeon3 kernel: [ 3452]     0  3452     2232        1   0       0             0 saslauthd
Jun  7 19:22:12 xeon3 kernel: [ 3453]     0  3453     2232        1   1       0             0 saslauthd
Jun  7 19:22:12 xeon3 kernel: [ 3454]     0  3454     2232        1   1       0             0 saslauthd
Jun  7 19:22:12 xeon3 kernel: [ 3455]     0  3455     2232        1   3       0             0 saslauthd
Jun  7 19:22:12 xeon3 kernel: [ 3456]     0  3456     2232        1   2       0             0 saslauthd
Jun  7 19:22:12 xeon3 kernel: [ 3467]   494  3467     3787      547   2       0             0 postgrey
Jun  7 19:22:12 xeon3 kernel: [ 3546]     0  3546     3327       26   0       0             0 master
Jun  7 19:22:12 xeon3 kernel: [ 3558]    89  3558     3596      105   1       0             0 qmgr
Jun  7 19:22:12 xeon3 kernel: [ 3576]     0  3576     1476       22   0       0             0 crond
Jun  7 19:22:12 xeon3 kernel: [ 3600]     0  3600    81829     2060   0       0             0 fail2ban-server
Jun  7 19:22:12 xeon3 kernel: [ 3691]     0  3691      502        1   0       0             0 mingetty
Jun  7 19:22:12 xeon3 kernel: [ 3693]     0  3693      502        1   3       0             0 mingetty
Jun  7 19:22:12 xeon3 kernel: [ 3698]     0  3698      502        1   1       0             0 mingetty
Jun  7 19:22:12 xeon3 kernel: [ 3700]     0  3700      502        1   3       0             0 mingetty
Jun  7 19:22:12 xeon3 kernel: [ 3702]     0  3702      502        1   3       0             0 mingetty
Jun  7 19:22:12 xeon3 kernel: [ 3704]     0  3704      502        1   1       0             0 mingetty
Jun  7 19:22:12 xeon3 kernel: [ 4672]    89  4672     3642       36   0       0             0 tlsmgr
Jun  7 19:22:12 xeon3 kernel: [29682]     0 29682     1230      211   1       0             0 config
Jun  7 19:22:12 xeon3 kernel: [13051]     0 13051      763        3   1       0             0 screenconnect
Jun  7 19:22:12 xeon3 kernel: [16817]   498 16817     1764       42   3       0             0 imap-login
Jun  7 19:22:12 xeon3 kernel: [16818]   498 16818     1764       43   3       0             0 imap-login
Jun  7 19:22:12 xeon3 kernel: [16825]  5000 16825     1059      128   1       0             0 imap
Jun  7 19:22:12 xeon3 kernel: [16826]  5000 16826     1043       48   2       0             0 imap
Jun  7 19:22:12 xeon3 kernel: [12269]     0 12269    18804      480   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19922]     0 19922     3047        3   0       0             0 sshd
Jun  7 19:22:12 xeon3 kernel: [19928]     0 19928     1378        3   1       0             0 bash
Jun  7 19:22:12 xeon3 kernel: [ 6403]     0  6107   767739   577098   1       0             0 mono
Jun  7 19:22:12 xeon3 kernel: [17953]    97 17953     3405      249   1       0             0 auth
Jun  7 19:22:12 xeon3 kernel: [18359]     0 18359     3405      260   3       0             0 auth
Jun  7 19:22:12 xeon3 kernel: [18362]  5000 18362     1023      115   3       0             0 imap
Jun  7 19:22:12 xeon3 kernel: [18610]    48 18610    33171    14047   2       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [18927]    48 18927    33147    14946   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [18998]    48 18998    29059     9753   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19002]    48 19002    32900    12509   2       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19138]    89 19138     3569      164   3       0             0 pickup
Jun  7 19:22:12 xeon3 kernel: [19181]    48 19181    33155    12180   1       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19252]    48 19252    33155    14846   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19305]    48 19305    32899    13820   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19393]    48 19393    28795    10018   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19394]    48 19394    33658    13610   1       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19395]    48 19395    28795    10309   3       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19488]    48 19488    33147    13654   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19491]    48 19491    28795     9598   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19492]    48 19492    29051    10418   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19516]    48 19516    29051     9323   1       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19557]    48 19557    28283     9483   1       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19558]    48 19558    29051    10229   1       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19678]     0 19678     3405      261   3       0             0 auth
Jun  7 19:22:12 xeon3 kernel: [19689]     0 19689     3405      261   2       0             0 auth
Jun  7 19:22:12 xeon3 kernel: [19793]    48 19793    28027     8690   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19812]    48 19812    28283     8522   1       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19850]    48 19850    28027     8388   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: [19880]    89 19880     3396      174   1       0             0 smtp
Jun  7 19:22:12 xeon3 kernel: [19881]    89 19881     3396      175   2       0             0 smtp
Jun  7 19:22:12 xeon3 kernel: [19883]    89 19883     3356      164   1       0             0 bounce
Jun  7 19:22:12 xeon3 kernel: [19901]    48 19901    27515     8268   0       0             0 httpd
Jun  7 19:22:12 xeon3 kernel: Out of memory: Kill process 6403 (mono) score 580 or sacrifice child
Jun  7 19:22:12 xeon3 kernel: Killed process 6403, UID 0, (mono) total-vm:3070956kB, anon-rss:2308240kB, file-rss:152kB


My "top" command, sorted by memory is below. Note that the server isn't anywhere near crazy busy with apache or other processes. That said, you can see mono is eating 1.9GB only 10 minutes of restarting the screenconnect services. This is insane, considering I'm not connected to any sessions. I have a grand total of 3 active "access" clients, and 7 "support" clients which have not even connected to their sessions yet.


top - 12:36:41 up 19 days, 22:21,  2 users,  load average: 1.48, 1.27, 1.18
Tasks: 231 total,   2 running, 229 sleeping,   0 stopped,   0 zombie
Cpu(s): 15.0%us, 11.3%sy,  0.0%ni, 67.2%id,  3.9%wa,  0.8%hi,  1.8%si,  0.0%st
Mem:   4019160k total,  3906876k used,   112284k free,   137216k buffers
Swap:   835580k total,   416164k used,   419416k free,   163228k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                    

                                                   
25662 root      20   0 2189m 1.9g 3156 S  0.0 49.1   0:49.64 mono /opt/ScreenConnect/Bin/ScreenConnect.Service.exe startservices 7 21424 10  
 3395 mysql     20   0 1015m 522m 3804 S 12.6 13.3   1754:49 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/li
29263 apache    22   2  130m  62m 5852 S  0.0  1.6   0:57.00 /usr/sbin/httpd                                                                 
 4714 apache    20   0  128m  58m 5856 S  0.0  1.5   0:44.16 /usr/sbin/httpd                                                                 
25616 apache    20   0  129m  58m 5684 S  0.0  1.5   0:14.33 /usr/sbin/httpd                                                                 
18557 apache    20   0  129m  57m 5848 S  0.0  1.5   1:05.69 /usr/sbin/httpd                                                                 
20237 apache    20   0  129m  57m 5884 S  0.0  1.5   1:02.57 /usr/sbin/httpd                                                                 
15004 apache    20   0  129m  56m 5760 S 17.6  1.4   0:32.22 /usr/sbin/httpd                                                                 
 3042 apache    20   0  128m  54m 5860 S  0.0  1.4   1:20.46 /usr/sbin/httpd                                                                 
24587 apache    20   0  128m  53m 5756 S  0.0  1.4   0:12.83 /usr/sbin/httpd                                                                 
14442 apache    20   0  112m  42m 5816 S  0.0  1.1   0:34.36 /usr/sbin/httpd                                                                 
30441 apache    20   0  112m  41m 6436 S 16.3  1.1   0:10.23 /usr/sbin/httpd                                                                 
30458 apache    20   0  109m  41m 6396 S  0.0  1.1   0:06.70 /usr/sbin/httpd                                                                 
30965 apache    20   0  109m  41m 6408 S  0.0  1.1   0:08.46 /usr/sbin/httpd                                                                 
29685 apache    20   0  111m  41m 6324 S  0.0  1.1   0:08.60 /usr/sbin/httpd                                                                 
30948 apache    20   0  109m  41m 6416 S  0.0  1.0   0:06.39 /usr/sbin/httpd                                                                 
29531 apache    20   0  112m  40m 6052 S  0.0  1.0   0:04.66 /usr/sbin/httpd                                                                 
30966 apache    20   0  110m  39m 6440 S  0.0  1.0   0:05.38 /usr/sbin/httpd                                                                 
  565 apache    20   0  109m  39m 6332 S 19.0  1.0   0:02.64 /usr/sbin/httpd                                                                 
31148 apache    20   0  109m  38m 6424 S  0.0  1.0   0:07.93 /usr/sbin/httpd                                                                 
27232 apache    20   0  109m  38m 5720 S  0.0  1.0   0:14.49 /usr/sbin/httpd                                                                 
29548 apache    20   0  109m  38m 5988 S  0.0  1.0   0:10.78 /usr/sbin/httpd                                                                 
 1118 apache    20   0  109m  38m 6336 S  0.0  1.0   0:02.37 /usr/sbin/httpd                                                                 
26820 apache    20   0  109m  38m 5768 S  0.0  1.0   0:15.87 /usr/sbin/httpd                                                                 
 1543 apache    20   0 97892  28m 5232 S  0.0  0.7   0:00.70 /usr/sbin/httpd                                                                 
 1406 named     20   0  111m  20m 1696 S  1.3  0.5 144:21.18 /usr/sbin/named-sdb -u named -4 -t /var/named/chroot                            
 3600 root      29   9  319m  10m 1580 S 33.2  0.3   1629:46 /usr/bin/python /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /
 2326 root      20   0 47296 6164 1592 S 11.0  0.2  12:02.03 /usr/bin/php /var/www/vhosts/infopackets.com/httpdocs/lists/admin/index.php -c/v
  664 root      20   0 13620 3960 2932 S  0.0  0.1   0:00.01 dovecot/auth -w                                                                 
 1489 opendkim  20   0 83020 3512 1040 S  2.0  0.1  71:57.05 /usr/sbin/opendkim -x /etc/opendkim.conf -P /var/run/opendkim/opendkim.pid      
 8598 root      20   0 13620 3280 2248 S  0.0  0.1   0:00.08 dovecot/auth -w                

                                     

Suggestions?


- Dennis

ConnectWise Control Version:
1
Server Affected:
Host Client Affected:
Guest Client Affected:

Wow, an entire month has gone by and still no response.


Since originally posting this issue, I completely uninstalled and reinstalled the product clean with a brand new configuration file and sessions database, and made only minimal changes to the configuration file. I've documented what I did here:


https://www.infopackets.com/news/10125/how-fix-uninstall-reinstall-screenconnect-server-mono-out-memory-error


Mono seemed to be behaving itself up until a few days ago, but now I'm right back where I was - in fact, the problem seems worse. Every click I make on a session eats 1 megabyte of memory. If I review a chat, another 300 megabytes gets eaten. If I review more than 3 chats I get "unknown error" in my browser and mono crashes and restarts on the server.


This is absolutely ridiculous. I've upgraded to the latest edition (ScreenConnect_6.3.13446.6374) and the problem has not been fixed!


I've sent an email to ScreenConnect sales and posted here, and have ZERO help from anyone. My license is going to expire on August 9th, 2017 and it seems I have little recourse but to keep reinstalling the product to "fix" the issue for a unknown duration of time until it creeps up again.


What can be done about this problem? How about connecting to my server and looking at it?




Waiting for information

Good morning,


It's not clear, based on the provided information, why the mono process is consuming all available memory.


Is it possible that the mono process on your server is running as a 32 bit process?


What is the output of running the following command as root on your server (insert the mono PID where indicated):


LD_SHOW_AUXV=1 ldd /proc/<insert mono PID here>/exe | grep AT_PLATFORM | tail -1


If the process is running as 64 bit, the output of the aforementioned command should be something like:


AT_PLATFORM:     x86_64



Regards,

Ben

Hello,


Thank you for looking into this. Yes, mono is 32-bit.


[root@xeon3 vmware:/var/log]# LD_SHOW_AUXV=1 ldd /proc/12596/exe | grep AT_PLATFORM | tail -1
AT_PLATFORM:     i686

According to this page, 32-bit Linux is supported for ScreenConnect server roles:


https://docs.connectwise.com/ConnectWise_Control_Documentation/Get_started/System_requirements#Linux


Thank you for confirming you're using a 32-bit implementation. I've so far been unable to reproduce the OOM behavior using a 32-bit install of CentOS 6.9 and Control 6.3.13466.


Please update this thread with the following info:


1) What is the size of your session database, located at App_Data\Session.db

2) How many access sessions are calling back to your server?

3) On average, how many concurrent active sessions (host and guest connected) does your server handle?

4) Have you made any customizations to your server (e.g.have you set Control to run behind nginx)?

5) What extensions have you installed?


Regards,

Ben

1) What is the size of your session database, located at App_Data\Session.db


-rw-r--r-- 1 root root 26M Jul 11 01:05 /opt/ScreenConnect/App_Data/Session.db


2) How many access sessions are calling back to your server?


Currently 4 Support sessions and 2 Access sessions.

3) On average, how many concurrent active sessions (host and guest connected) does your server handle?


Usually I only connect to 1 user at a time but sometimes I use 2 concurrent sessions as that is the maximum that my license can handle.


4) Have you made any customizations to your server (e.g.have you set Control to run behind nginx)?


I run: Drupal, Apache, Dovecot, PhpMyAdmin, ScreenConnect, Perl, PHP, Mysql, Fail2ban on the server primarily. I don't use nginix.


5) What extensions have you installed?


None.


Note the screen captures below which show a whopping 300mb RAM jump in only 1 minute time. All I did was click on 3 conversations from clients and clicked on their sessions.


You are more than welcome to connect to my server to review the issues if you cannot replicate it.






Hello,


As I mentioned previously: after running out of memory repeatedly (about a month ago), I've reinstalled ScreenConnect - but it wasn't until weeks later I've come into this problem once again. If you cannot reproduce the issues on your own virtual machine, then how about SSH'ing into my system to have a look around? This does not seem to be an unreasonable request.


I have had this memory leak issue for at least 4-6 months now. Upgrading to CentOS 64bit to fix this problem doesn't seem like a legitimate answer since (a) that would require reconfiguring my entire server which took weeks to set up; (b) your support page says Linux 32 bit is supported, and (c) there is no guarantee the issue will be fixed since you don't really know what the bug is in the first place.


- Dennis


Dennis,


Did you ever find the solution to this issue? I am experiencing the exact same thing on my server as well. I am running a 2GB VPS over at OVH and I have noticed this issue happening for a while. It gets to the point where I cannot even load SC.

No I did not find a solution to this. I was more or less told that the 32-bit version of Mono (part of screenconnect) was unpredictable and that upgrading to 64-bit OS was recommended. That said I have not had an 'out of memory' issue for quite a while now. You can try and reinstall screenconnect as that might help. Also, from what I have read, screenconnect seems to be fairly stable in a Windows environment if you have access to that. I have not tried, so I can't say.

I am running the 64 bit version and having the same problem...


I was debating about launching a Windows VPS over at OVH and installing it. I just have to deal with the issue of the moving 70+ computers.