VMware Server Hangs on ‘Mount ISO’ Browsing smbmount

Hahaha. So I’ll try to keep this brief, but I just wanted to post the problem I recently found with VMware Server (1.0.4, 1.0.5) on linux (perhaps others), and what I eventually found to be the cause and workaround.


Running VMware server on linux, I edit virtual machine settings, open the CD-ROM device, select Use ISO image, and Browse. I navigate to my smbmount (actually a cifs mount line in my /etc/fstab), and VMware hangs for about three or four minutes before displaying the contents of the directory. The following lines are written to /var/log/syslog:

Mar 24 16:54:30 nine kernel:  CIFS VFS: server not responding
Mar 24 16:54:30 nine kernel:  CIFS VFS: No response to cmd 47 mid 16
Mar 24 16:54:30 nine kernel:  CIFS VFS: Write2 ret -11, written = 0
Mar 24 16:55:30 nine kernel:  CIFS VFS: server not responding
Mar 24 16:55:30 nine kernel:  CIFS VFS: No response to cmd 47 mid 28
Mar 24 16:55:30 nine kernel:  CIFS VFS: Write2 ret -11, written = 0

Tracking down the problem

I started by just googling these errors for a while, with no particular luck. The closest solution I found was someone getting errors like this, and eventually resolving them with a new NIC and/or switch … some network problem. I was hopeful this might be my problem, since I was connecting at 100/Half through an old hub (not switch) on all machines exhibiting the problem (though, they did have various types of NICs, so I didn’t think it was a driver bug). Unfortunately, even after I was able to get a switch instead of a hub, the problem persisted. The #samba IRC channel was no help, and I wasn’t ready to bother the developers.

I put the problem aside for a few days, and gave it another shot on a clean state of mind. This time, I used lsof to figure out which pid had an open handle on my smbmount. I then attached strace to the process and tried to open a directory, and was greeted with this little gem (comments added by me):

/* create .vmBigFileTest0 for LARGEFILE access */
open("/blah/blah/.vmBigFileTest0", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 36
/* override the regular signal handler for 'exceeds file size' with an
 * 'ignore signal' handler
rt_sigaction(SIGXFSZ, {SIG_IGN}, {0x80c9f00, ~[KILL STOP RTMIN RT_1], SA_RESTART}, 8) = 0
/* set the read/write offset to 2 GiB */
_llseek(36, 2147483648, [2147483648], SEEK_SET) = 0
/* write a byte */
write(36, "\\245", 1)                    = 1
/* close ... which also means flush buffers to 'disk.'
 * this call hangs for a long time
close(36 <unfinished ...></unfinished>

Download this code: vmware-strace.c

To summarize: VMware apparently thinks it’s necessary to check if the underlying filesystem of its ISO mount location has a 2 GiB (or smaller, I suppose) filesize limitation. While this is a very legitimate concern when browsing for the path to create virtual disk files, it strikes me as pretty irrelevant for mounting an ISO, which should be a read-only operation. To make matters worse, it doesn’t do this by checking the filesystem type in /proc/mounts or something quick and easy like that … it does this by effectively writing 2 GiB of data. (In fairness, their way is more reliable and future-proof, but come on…)


Once I saw what was going on, the solution to this problem was pretty trivial. I remounted my smbmount to be read-only, rather than read-write, which prevents VMware from getting the chance to try to write 2 GiB of data. In case you really don’t want to do any more research yourself, this means either passing the -o ro option to the mount command, or adding ro to the comma-separated options list of the fstab entry (right after the fs type entry … i.e. cifs).

2 Responses to “VMware Server Hangs on ‘Mount ISO’ Browsing smbmount”

  1. Gloria Says:

    Brilliant! Thank you. I was wondering why it’s hanging here.

  2. Nicolas Says:

    Thanx for your post that opens a way (maybe) to explain my problem, which is slightly different from yours but with same effects:

    I am running vmware server (1.0.5) on linux ant I am doing a copy of data from the linux to a windows (virtual machine) 2003 server every day by a cron job. The copy is done by the following steps:

    1- Mounting a windows share on the linux machine
    2- Coping the data
    3- Unmounting the windows share (leaving the mount would freeze the machine every 2 or 3 days!!)

    Than, I have the linux system that “freezed” every 4 or 5 days and exactly the same messages in /var/log/syslog.

    So maybe it isn’t just the “Mount ISO” browsing smbmount bug, but a hole smbmount problem with vmware ….

    Knowing this “possible” issue, I will try to disable this script and see if the machine will stand more days.


Leave a Reply