Wednesday, July 18, 2007

VMWare : Loosing eth0 after you've copied your VM

Background

Here's a bit of weird-behavior I've noticed when working with some of our production virtual machines (running Gentoo Linux) here at work.

In order to update the OS's on our virtual machines, I will copy them to my local machine, power them on, update them, and then push the updated OS's back out into production at the earliest convenience.

When you copy the VM from one location to another, VMWare notices this and asks you "Hey, it looks like this machine has been physically moved or copied, do you want me to create a new VM-UUID?" If you answer in the affirmative, VMWare internally regenerates any unique-identifiers tied to this virtual machine. The one thing that's really noticeable is that any virtual ethernet adapters get their MAC addresses changed.

The problem I've experienced is that when you power on the new-UUID'd VM, you no longer have an ethernet adapter. Gentoo tries to bring-up eth0 and it says "network interface eth0 does not exist" and "Please verify hardware or kernel module (driver)"


Explanation

"So, what's going on?"

Try a couple things:
  • If you run lspci you should still see the ethernet adapter.

  • If you run 'dmesg' and should see the kernel find the network card and it even calls it eth0


"So, where does eth0 go?"

Try running ifconfig -a. I bet you now have an eth1 and it's MAC address matches the newly-generated virtual MAC address specified in the virtual machine's .vmx file.

"Oh great, so every time I copy the VM I need to update the system configs to use the new eth1, or eth2, etc!?!?!"

No, hush, I'm getting to the answer.

The problem stems from the linux distro 'remembering' the MAC address of the network adapter and expecting it to be the same between boots. In the case of our Gentoo VM's, it's udev that mucks this up.

"Ok, fine, it's udev's fault. We know it's broken because it's expecting the ethernet adapter to have a MAC address that it no longer has. What to do?

The Answer

To fix this problem you need to tell your linux distro the VM's new MAC address. How you do this can vary by distro. In my spelunking, I found a few ways:

  • In Gentoo do one of the following: (Do #1, it's the easiest.)
    1. Delete /etc/udev/rules.d/70-persistent-net.rules and reboot. Your eth0 should be back.
      • 2007/09/13 Update: This almost-always works for me. But, for some reason, sometimes it seems to confuse udev even more; after rebooting, I'll have an eth2 or eth3. When this happens, I end up following #2, making sure the udev config file has 'eth0' listed, and not eth1, eth2, or eth3.
    2. Edit /etc/udev/rules.d/70-persistent-net.rules (or whatever it's named) to match your new MAC address and reboot. Your eth0 should be back.
  • Other distros:
    • Look for, (and edit if you find,) /etc/iftab
    • Look for, and delete, then reboot /etc/udev/rules.d/25-iftab.rules
    • Look for, (and edit if you find,) /etc/sysconfig/network-scripts/ifcfg-eth0



Give Credit Where Credit Is Due

I got hints from a number of pages, but in the end, it was the folks over at the VMWare discussion forums for the win:
VMWare Discussion Forums

32 comments:

Anonymous said...

Neh! Hahaha, you may just be one of the biggest nerds I know. Not that it's a bad thing. I almost thought your pony tail wasn't long enough to solve the problem. ;-)

Jason Poll said...

Hey, I don't have this geek bumper sticker for nothing, you know?

My usual philosophy for blogging 'geek content' is, I'll talk about things that give me trouble that weren't almost-immediately solved 'by google'. I had to dig a bit to eventually get to the solution mentioned in this post, and figured, "I should share this."

Anonymous said...

Thanks Jason,

I just copied a VM of Deki Wiki that I have been messing with locally, up to our corporate server for a demo tomorrow and eth0 was indeed down. This just saved my arse!

This was a Debian distro and I used option #1. The file was z25_persistent-net.rules in my case but worked just the same.

Thanks a bunch.

Jason Poll said...

Richard, glad to hear this advice helped someone else.

Anonymous said...

Thanks for the write-up:
1) problem...
2) google
3) your write-up
4) ...solved

3 minutes :^)

Anonymous said...

This may be "geek content" but it has helped me the same with a non-working Deki Wiki installation. Thanks!

Anonymous said...

Thanks for posting the solution, and making it available through Google. I ran across this problem a while ago...didn't remember the exact fix and needed it again today.

Anonymous said...

excellent writeup on the issue. the problem got posted several times on the vmware forums as well.
thanks.

Anonymous said...

Thanks so much for the tips! I was stuck till I saw your write up.

Anonymous said...

Thank you very much for your help. The solution was exactly what I was looking for.

Anonymous said...

You just saved me a few more grey hairs. Thanks

Anonymous said...

Perfect, great solution for that issue i just had a minute ago.

Rocker said...

You just saved me from data loss and having to redo 8+ hours of work. Thank you!

Anonymous said...

Just another happy netizen thankful for your writeup. Sure enough I 'lost' my eth0 and udev was the culprit.

Its a little sad to see this misfeature in Linux, til now I'd only been aware of it in Windows.

Jason Poll said...

Steve: Indeed this is sad to see, but I think I can imagine the logic behind what udev is doing. If we weren't dealing with a virtual machine, the only time udev would be seeing a new MAC address would be if you add a new NIC to the machine.

Now, maybe udev should say: "Wait a minute, is the old MAC/NIC that I have associate with eth0 still in the machine? If not, I bet this new MAC/NIC is the new eth0."

But hey, I'm not aware of the udev design, so I'm just guessing. :)

Anonymous said...

Thanks! This worked for me with a naughty Slackware 12.1 VM that had forgotten it's MAC and lost eth0.

Cheers, Si. :)

Anonymous said...

Thank you very much, this helped me out while I was playing with VMware Server and an Ubuntu 8.04 guest install that I copied to a remote USB hard drive. I was trying to see if it was possible to copy to an external HD for acessing on other systems via USB (portable linux install). After the copy I lost eth0 and only after reading your post was I able to sort the issue by manually editing /etc/udev/rules.d/70-persistent-net.rules. Thanks for posting the solution!

Anonymous said...

Thow art the man.
Well done , nice post
Thx A Million.

Anonymous said...

Thank you so much. I've been terribly frustrated by this problem. I deleted the 70-persistent-net.rules file and now have internet connection again.

Anonymous said...

Thanks a lot.
Info: With Debian 4.x the file has another number in its name and you have to manually change eth1 to eth0. The file is recreated with eth1.
Again ... THANKS :-)

Unpec said...

Good job, thanks a lot for solving my problem

林阿喵的幻想 said...

It works
Thank you :D

Anonymous said...

Hit this problem, this was the first google link.. solved it straight away.. saved me ages :)

Unknown said...

Worked for me, thanks a lot!

ilja said...

yea, that solved it. thanks !

Helen said...

http://helenofdestroy.tumblr.com/post/815880725/yoopergeek

viagra online said...

Interesting article, added his blog to Favorites

Sreejith_Gireesan said...

Fantastic - you saved my day!

Anonymous said...

Thanks! This also works for REL 6.x

Anonymous said...

worked for me.
thanks!

Anonymous said...

Thank you man !

Anonymous said...

Thanks!!!

you saved me!!!!