vSphere 5.1 and VMs Inaccessible After NAS downtime

What it looks like

We start with an email I found in my Inbox, stating that my Nexenta-VM had a problem and was rebooted at 1:26AM. After the reboot all the shares became available again, but my vSphere environment was left in a mess:

vCenter showing inaccessible VMs after a short NAS failure

vCenter showing inaccessible VMs after a short NAS failure

As you can see a lot of VMs are inaccesible. Actually only VMs that were powered off at the time show up greyed out; all VMs that were running during the NAS failure have been hanging, but they resumed properly after the NAS came back online. In the example above I actually already fixed some of the VMs; originally ALL powered-off VMs were in the “inaccessible” state!

Proof that they are not inaccessible

First off I browsed to one of the inaccesible VMs on the NAS datastore, and found that all seemed well:

Even though the vSphere 5.0 node claimed the VM was "Inaccessible", all seems well at the datastore level...

Even though the vSphere 5.0 node claimed the VM was “Inaccessible”, all seems well at the datastore level…

I tried to get some VMs back the “hard” way:

  1. Remove the VM from inventory;
  2. Browse for the VM on the NAS datastore;
  3. Register the VM again using the *.vmx file.

This approach does actually work, but it leaves you with a “new VM that is an old one”. So VMware View… Won’t get it. Your backup appliance… Won’t get it. It just takes more fixing than you might anticipate. So I came up with a way simpler approach…

Fixing it the easy way

I figured I needed to somehow tell vSphere that the inaccessible VM was actually no longer inaccessible, and force it to reload the configuration. And vimsh can do exactly that… It has the ability to reload a VM into a host, and exactly that does the trick!

First, we need to access the host using ssh or the direct commandline as root. From there, you can simply find all VMs that are inaccessible using the following command:

vim-cmd vmsvc/getallvms >grep skip

As an example, this will generate output like this:

~ #
~ # vim-cmd vmsvc/getallvms >grep skip
Skipping invalid VM '118'
Skipping invalid VM '127'
Skipping invalid VM '147'
Skipping invalid VM '16'
Skipping invalid VM '184'
Skipping invalid VM '185'
Skipping invalid VM '190'
Skipping invalid VM '25'
Skipping invalid VM '92'
Skipping invalid VM '94'
Skipping invalid VM '95'
Skipping invalid VM '97'

These are actually all the VMs that are inaccessible on this host right now. Now it is easily fixed, by simply calling a reload for each “skipped” VM using this command:

vim-cmd vmsvc/reload [NUMBER]

This will trigger reload actions on the host:

Reloading the entities as a result from reload commands and presto! The VM is accessible again.

Reloading the entities as a result from reload commands and presto! The VM is accessible again.

So simply repeat the reload command for all the “skipped” VMs and your problem is solved. Should be easy to script in Powershell as well.

NOTE: The numbering between the VI client and vimsh actually DO NOT MATCH (see example above). So make sure you use vimsh to find the numbers of the inaccessible VMs, and not use the VI client for this!

To my understanding this is a known issue: VMware plans to fix this in vSphere 5.1 update1. In the mean time, use the reload described here and avoid removing and re-adding your VMs.

Source: http://vmdamentals.com/?p=4503

Advertisements

How to Build OpenVPN Server on CentOS 6.x

How to Install, Setup, Config OpenVPN on CentOS 6.4 – In this page I write full tutorial to guide you installing OpenVPN on CentOS 6.x server. I will try all the steps to be as clear as possible. Do not hesitate to ask if you have any question. Previously: How to install PPTP on CentOS 6.x (the easiest way).

What you need?

  1. A VPS or Dedicated server running CentOS 6.x
  2. Proper knowledge to use Putty, SSH and common Unix command
  3. Only for VPS based-on OpenVZ virtualization (other skip this): please enable TUN/TAP options in your VPS control panel (e.g: SolusVM).

OpenVZ VPS users only:

enable tun tap ppp

How to Install OpenVPN to Build CentOS VPN server

Prerequisite

Step 0 – Login to your server via SSH. You better login as root.

Step 1 – Now issue this first command syntax:

1
yum install gcc make rpm-build autoconf.noarch zlib-devel pam-devel openssl-devel -y

screenshot:

add repo

Step 2 – Now download LZO RPM and Configure RPMForge Repo. Use wget command:

1
wget http://openvpn.net/release/lzo-1.08-4.rf.src.rpm

screenshot:

download repo

Step 3 – Now add correct repo for your server:

CentOS 6 32-bit (x86):

1
wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-1.el6.rf.i686.rpm

CentOS 6 64-bit (x86_64):

1
wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

screenshot:

add repo

How to know which one is your server? Issue this command:

1
uname -a

If you see “x86_64 GNU/Linux” at the end of the output line means your server is 64-bit. Otherwise if you see “i686 i386 GNU/Linux” or “x86 GNU/Linux” means your machine is 32-bit.

vps architecture

Step 4 – Then build the rpm package using this command:

1
2
3
rpmbuild --rebuild lzo-1.08-4.rf.src.rpm
rpm -Uvh lzo-*.rpm
rpm -Uvh rpmforge-release*

hit enter for each line above.

rpm

Installing OpenVPN

Step 5 – Issue the special yum command:

1
yum install openvpn -y

screenshot

install openvpn

Step 6 – Copy the easy-rsa folder to /etc/openvpn/, use this command:

1
cp -R /usr/share/doc/openvpn-2.2.2/easy-rsa/ /etc/openvpn/

Step 7 – Now edit it:

1
nano /etc/openvpn/easy-rsa/2.0/vars

Edit this line:

1
export KEY_CONFIG='$EASY_RSA/whichopensslcnf $EASY_RSA'

replace it with:

1
export KEY_CONFIG=/etc/openvpn/easy-rsa/2.0/openssl-1.0.0.cnf

screenshot:

edit rsa

once done hit Control+O to save then Control+X to exit.

Step 8 – Create the certificate using these commands:

1
2
3
4
5
cd /etc/openvpn/easy-rsa/2.0
chmod 755 *
source ./vars
./vars
./clean-all

hit enter for each line.

rsa config

Step 9 – It’s time to build necessary CA file:

1
./build-ca

screenshot:

build ca

Hint

Country Name: may be filled or press enter
State or Province Name: may be filled or press enter
City: may be filled or press enter
Org Name: may be filled or press enter
Org Unit Name: may be filled or press enter
Common Name: your server hostname
Email Address: may be filled or press enter

Step 10 – Time to build Key Server:

1
./build-key-server server

screenshot:

sign certificate

Hint:

Almost the same with ./build.ca but check the changes and additional
Common Name: server
A challenge password: leave
Optional company name: fill or enter
sign the certificate: y
1 out of 1 certificate requests: y

You can simply leave them blank. The only 2 required are sign the certificate (choose “y”) and 1 out of 1 certificate requests (choose “y”)

Step 11 – Now issue command below to build Diffie Hellman:

1
./build-dh

screenshot:

build dh

Step 12 – Create OpenVPN config file:

1
nano /etc/openvpn/server.conf

Step 13 – Now enter this value in that config file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
port 1194 #- port
proto udp #- protocol
dev tun
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
reneg-sec 0
ca /etc/openvpn/easy-rsa/2.0/keys/ca.crt
cert /etc/openvpn/easy-rsa/2.0/keys/server.crt
key /etc/openvpn/easy-rsa/2.0/keys/server.key
dh /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem
plugin /usr/share/openvpn/plugin/lib/openvpn-auth-pam.so /etc/pam.d/login #- Comment this line if you are using FreeRADIUS
#plugin /etc/openvpn/radiusplugin.so /etc/openvpn/radiusplugin.cnf #- Uncomment this line if you are using FreeRADIUS
client-cert-not-required
username-as-common-name
server 10.8.0.0 255.255.255.0
push "redirect-gateway def1"
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"
keepalive 5 30
comp-lzo
persist-key
persist-tun
status 1194.log
verb 3

Save it once done. (Control+O then Control+X)

ovpn config

Step 14 – Lets start OpenVPN service on your server for the very first time:

1
service openvpn start

pic:

start ovpn

Step 15 – You’ll also need to enable IP forwarding in the file /etc/sysctl.conf. Open it and edit “net.ipv4.ip_forward” line to 1:

1
nano /etc/sysctl.conf

replace 0 with 1 in this line:

1
net.ipv4.ip_forward = 1

pic:

ovpn13

Hit Control+O to save then Control+X to exit nano.

Step 16 – Issue this command to load the change:

1
sysctl -p

Step 17 – Create new Linux username which can also be used to login to the VPN:

1
useradd username -s /bin/false

replace username with your own username.

Then also create its password:

1
passwd username

pic:

ovpn14

Step 18 – Now route some iptables.

Xen and KVM users use:

1
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

special for OpenVZ use these two instead:

1
iptables -t nat -A POSTROUTING -o venet0 -j SNAT --to-source 123.123.123.123

and

1
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source 123.123.123.123

Do not forget to replace 123.123.123.123 with your server IP. Pic:

ovpn15

Step 19 – Note: if you have CSF on the same server you need to open your OpenVPN port (Usually 1194) through the firewall and run the below commands for CSF:

1
2
3
4
5
iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -s 10.8.0.0/24 -j ACCEPT
iptables -A FORWARD -j REJECT
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -j SNAT --to-source 123.123.123.123

Step 20 – Now save that iptables rules:

1
service iptables save

Step 21 – Finally lets create a server.ovpn config file. To make it easy, you can simply create it on your local computer using Notepad (or any other simple text editor tool). Enter following in that file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
client
dev tun
proto udp
remote 123.123.123.123 1194 # - Your server IP and OpenVPN Port
resolv-retry infinite
nobind
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
ca ca.crt
auth-user-pass
comp-lzo
reneg-sec 0
verb 3

Then save it with .ovpn extension. Save that file in the config directory of where you installed OpenVPN client in your computer. See screenshot:

ovpn config file

Step 22 – That’s it. Now you can copy ca.crt file from /etc/openvpn/easy-rsa/2.0/keys/ directory and place it in your server’s document root (public_html).

1
cp /etc/openvpn/easy-rsa/2.0/keys/ca.crt /path/to/public/directory

example:

1
cp /etc/openvpn/easy-rsa/2.0/keys/ca.crt /var/www/servermom.com/public_html

Now you can download the ca.crt file from your browser by going to domain.com/ca.crt then save it to the same folder as .ovpn file you created earlier.

 

Source: http://www.servermom.com/how-to-build-openvpn-server-on-centos-6-x/732/