Category Archives: SysAdmin

Comments up and running again

Just a brief public service announcement. After being made aware over the weekend that the comment system wasn’t working [h/t to Jeff] I’ve figured out what happened. There was a collision between the captcha component and the single sign-on component in WordPress. It’s all ironed out now and you should be able to pontificate to your hearts desire.

Reminder that due to spam volume it’s still moderated comments, but I try to push them through quickly.

SSH Tips – How to Login Securely Without a Password

old fashioned keys SSH is a wonderful tool and will let do do all kinds of amazing things – not to mention that it does them securely. However sometimes when you’re trying to automate steps, or are performing the same steps repeatedly on a trusted machine, the frequent retyping of your username can be a pain. Worse still, if you’re writing a script, you certainly don’t want to hardcode passwords into it for others to grab. In this case, what you can do is use ssh keys to secure your connection.

How to do this differs depending on the operating system of the source machine, IE the machine you are SSHing from. Suppose you have two machines, the local one (your laptop) and the remote one (some server, eg my.server.com) To ssh from the laptop to the server without needing a password, perform these steps:

Linux

On the local machine:
% ssh-keygen -t rsa
Either put in a passphrase or just hit return twice to skip. Note that using a passphrase makes it more secure, but makes automation tricky.

This produces a file called id_rsa.pub in a subfolder called .ssh underneath your some directory. Now you need to transfer that file to the remote server. Note that you’ll need your password to perform this step, and to avoid troubles we’ll rename the file during transfer.

% scp ~/.ssh/id_rsa.pub USERNAME@my.server.com:id_rsa.pub.mylaptop

Now we need to add the id_rsa.pub keys to the proper file on the remote machine (my.server.com). Note that if you don’t already have a .ssh folder on the server, you can just create it, or better yet, run the ssh-keygen command there, as above.

% ssh USERNAME@my.server.com
% cd .ssh
% cat ~/id_rsa.pub.mylaptop >>; ~/.ssh/authorized_keys
% rm ~/id_rsa.pub.mylaptop

Make sure .ssh dir and all it’s files don’t have any open group or other permissions, or this won’t work.

% cd ~/.
% chmod -R go-rwx .ssh

Here’s an example of what the authorized key file will look like. Note that there will be one line (word-wrapped) per each user/machine that has exchanged keys in this manner.

%cat ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAyJnwH/k4/FdY88p2utHHDc5VSJqL97n/nsK1PkW9q9KWddMIu8u+Golyg4RW10nIGs3A4EYYPn9Gu7dJy+vhO2xRJM4A+EEF/9nYYy/ZLXBlh4V3zMRYLom6TZx9OSTA6L0z9HKdopgJ/HnQ+yEFzS29TBjCs/9Dy4+iS0uVhWs= root@krysia.afraid.org

Windows

Use your favorite ssh tool and check it’s documentation. For now I’ve used puttygen, it should be where you installed putty, probably something like c:\bin. It is a graphical program for managing keys on windows with putty.

Select “SSH2 (RSA)” as the type of key (at the bottom of the screen)

Select “generate” and follow the instructions. It wants you to move your mouse around in a block for awhile to generate randomness. Then it makes a key.

Select “save private key” and either give it a passphrase, or ignore it when it tells you to think about using a passphrase. you can save your private key to disk somewhere. Note that using a passphrase makes it more secure, but makes automation tricky.

Select “save public key” and save it to disk somewhere.

In the normal putty window select load to pull in the profile you want to add the key to. Go to Connection and put the ID in the “auto-login username” box. IE your unix login name.

In the SSH Auth section select the browse button to go to where you stored the private key file, and select it. THen go to the “session” category and select save.

Now you need to take the public key stuff and add it to your ~/.ssh/authorized_keys file on the ssh server machine. If you’re using putty you have pscp that you can use. It’s in the same dir where you put your putty executable.

c:\> cd dir_with_public_key_file
c:\> pscp putty_public_key_file USERNAME@my.server.com:id_rsa.pub.mylaptop

Now connect to the remote system using ssh so you can add your public key to their authorized keys file, IE use ssh or putty. After you’re connected, edit the file you put there, id_rsa.pub or whatever you called it.

Remove the first line of the file that says “BEGIN SSH2 PUBLIC KEY”

Remove the last line of the file that says “END SSH2 PUBLIC KEY”

Remove the line that says “Comment: ”

At the beginning of the first line insert “ssh-rsa ”

At the end of the last line after the =, put something that says what the key is for future reference. IE your user/machine name, like this, instead of “=” put “= user@machine.company.com”

Now there are probably 4 lines in this file, and they all need to be joined into one line. Plus if joining creates spaces they will need to be removed.

Now you can append this to the ~/.ssh/authorized_keys file:

% cat ~/id_rsa.pub.mylaptop >> ~/.ssh/authorized_keys
% rm ~/id_rsa.pub.mylaptop

As an extra bonus, if you’re trying to use the pscp command line in windows (it’s the windows equivalent for scp in unix) then here’s how to do it.

Make sure you’ve done the public key transfer, as above. Then when you call pscp, just pass the “-load” option with the name of the “profile” that you’re using.

Hopefully this helps – I find it very useful. If there are other operating systems, or other tips you’d like to know, just ask.

Virtual Machine Networking Options

3Q12 10:01 Derby RTC - Carlisle Wapping Sidings

I am frequently asked to help people setup virtual machines. One of the most common questions is about how to setup networking properly, so I thought I’d share that information here.

Virtual machines have a variety of networking setups available. In addition there are also many options for the underlying OS, such as Windows, OSX, or Linux. Before you can make real use of a virtual machine, it will be necessary to make sure that you have a proper IP address. Enterprise deployments should be on a dedicated hypervisor, such as
vSphere / ESXi from VMware, VirtualBox, Hyper-V, etc. For them you setup networking as you would any other machine.

The VM can use either a static IP address or DHCP. By default virtual machines are often preset for DHCP or dynamic setup. If you want to use a static address you will need to set it to an appropriate value. For more details on linux networking, check the Ubuntu Network Configuration guide.

Desktop VM Network Configuration
VMware provides three types of networking configurations: Nat, Bridged, and Private or host-only. I expect most desktop hypervisors to have similar options.

NAT addressing
NAT means that the VM is running on a private network confined to the host machine. It can see external machines, but is generally unavailable to other machines on the network – they cannot see it. The IP address for the VM is supplied from the VM server itself, using DHCP. This setup is well suited to normal, personal use or a demo, but is usually inappropriate for an actual deployment.

Bridged addressing
Bridged lets your virtual machine share the network adapter from the host machine. In
this setup, your machine gets a “normal” IP address on your network. Typically this
is something like DHCP, and may require you to give the MAC address of the VM to your system administrator. This is a good setup if you need others to be able to access the VM. You can find your MAC address by following the instructions above under “Getting the MAC address”. If you know the correct range of IP addresses, you can also assign a fixed address this way, but you’ll need to have all the necessary information for a static network configuration.

Host-only addressing
Host-only mode lets your VM ONLY see your host machine. The VM has no access to
the network or the world at large. This is ok for certain demos, but not useful for general purpose use. It is very secure.

Your Address
You can find your network address and mac address either by using the ifconfig command from a prompt for linux, (described above under “Getting the MAC address”), or from the user menu when logged in you can select “System – Preferences – Network Connections”. For windows you can using the ipconfig command or right-click your networking icon.

Getting the MAC address
If you want to use DHCP for the address, you will probably need to give the MAC address for the VM to your system administrator. Some networks may require the MAC address for any access at all. From the VM terminal (see “logging in” for details) you can run the command: “ifconfig”. You will see two or more entries. One is “lo” which is for the loopback and can be ignored. The other should be eth followed by some number, such as eth1 or eth2. Take note of this value as you will use it later on in the configuration. On the same line as the ethx you should see a label that says “HWaddr” – the value following that label is your mac address for that network interface.

If there are others tricks you know, please share them. If there are other topics you’d like me to cover, let me know.

Java Memory Settings – JVM Heap Size

Haystack © by bengal*foam
When you run a Java program you don’t execute it the way you do an executable written in languages like C/C++. The program must run inside the Java virtual machine or JVM, which interprets the compiled Java byte code and translates it to the local native instruction set. For example, to execute a Java program called HelloWorld you would type:

java HelloWorld

The amount of memory available to the application is known as the Java heap size and is controlled by passing the -Xms and -Xmx flags to the virtual machine. The Xms is the starting heap size, the Xmx is the maximumum heap size. On machines with resource contraints you can set them to a small starting size, and a max that is large enough to do what you want to do, like “-Xms128MB -Xmx512MB”. This will free up more memory for other applications, but does impact performance, as I’ll discuss more in just a moment.

If you don’t pass the flags, you get whatever the default settings are, typically something like 32MB for starting and 128MB for max – but it depends on your OS and the particular implementation of Java you’ve installed. For example if you want to set the max heap size to 256 GB you can type:

java -Xmx256gb HelloWorld

However, you can’t just put any number you want there and have it work. It is a known fact that 32-bit architectures limit the amount of memory any single process can allocate to 4GB. So why does the JVM limit the maximum heap size to something much less than 2GB?

When you try to run your JVM with a -Xmx2gb flag, you’ll get the following error:

Error occurred during initialization of VM
Could not reserve enough space for object heap

This is a limitation of a 32-bit OS such as Windows XP. 32-bit processes can only use a maximum of 4GB memory address space. Windows further splits that into half by allocating 2GB to the kernel and 2GB to the application. The reason you can not hit the 2GB limit within the VM is because there is memory overhead that the VM and OS use for the process, hence you end up with a bit less than the actual 2GB limit. The solution would be to move to a 64-bit machine running a 64-bit OS, and a 64-bit JVM.

In tests I have found the overall max you can use with a 32-bit JVM is OS/JVM dependent. Typically on Windows it’s about 1696MB, Linux is about 1696MB, and on Solaris it’s about 3000MB.

I’ve gotten the chance to work with a large variety of java applications on a wide variety of machines as part of my day job at Parasoft and I’ve found some configurations that will make things run better.

On a server machine I recommend you set the min and the max to the same (upper) value. This has a couple of benefits. First, it’s generally faster because you don’t lose any time for the JVM resizing. In addition, in cases where you aren’t completely sure that the meory will be available, this let’s you know immediately that the program doesn’t have enough memory, because it will refuse to even start. You can test this by starting enough applications that you’re down to less than 1GB of free memory, then typing:

java -Xms1024m -Xmx1024m HelloWorld

and it will fail again with a heap error because it is unable to allocate the 1GB of memory we told it we needed. This is useful with servers because we avoid confusing out-of-memory error messages when we thought we had enough but didn’t.

For performance issues, max memory should be 500MB less then your total real memory on Windows and 200 MB on linux/unix. For example if your Windows machine has 1GB of ram, the max setting should be set to 512 MB.

An additional note on contiguous memory. The JVM handles it’s memory as a single contiguous block. This means that if you give the JVM different parameters for initial (-Xms) and max (-Xmx) then the size of the JVM may change dynamically while running. However, this has some frequently unexpected ramifications.

The first is performance. If you have -Xms256m -Xmx1024m then your JVM will start with
an initial 256MB. If you’re running along happily and then you need something more like 300MB, here’s what happens:

  • First the JVM allocates a completely new block of 300MB
  • Then there is a transfer (memcopy) of the existing memory
  • then the original 256MB block is released.

As you can imagine this can be an expensive operation. You may have expected that it would simply go find an available block of 44MB and allocate it, but because of the JVM needing contiguous memory, there is no other way to handle it. This means the expansion tends to be slow. This is fine on machines with limited memory, but on large machines you can achieve much better performance.

A second effect is that you can get unexpected crashes if the system is unable to make the
allocation for the new block. For example, let’s assume you have a machine with 1GB physical memory and 1GB of swap. You’re running “normally” using about 600MB of memory and want to start a Java app that uses a lot of memory. You give the JVM -Xms512m -Xmx1024m and it starts fine.

Obviously there is a bit of swapping going on, but you’re running and happy. Then at some point your app starts needing more memory, and it wants to use the full 1024m that you’ve told the JVM is available. To review, you are currently using 600MB (stuff) + 512MB (java app) out of 2048MB (total memory). Now if you want to allocate your JVM from 512 to 1024, you would think that you only need an additional 512, but that is incorrect.

What actually happens is that you need a new block of 1024 to be allocated, AFTER which
the original memory is freed. Typically when the new block is large than the OS has available, the JVM is unable to handle it cleanly and crashes. This is a perfect example of how a JVM can crash even though you thought you had enough memory. In this case, setting the
parameters to -Xms1024m -Xmx1024m on the same machine would have succeeded, even
though you might think it was less likely than the smaller initial allocation.

Because of this, I normally recommend that if you have performance issues, or unexplained
crashes, to try and figure out how much memory you really need, and set Xms and Xmx to the same size. For typical Java apps, this can be 256 or 512 or even less. For large intensive apps 1024 is a better choice.

If you’ve got other tips or more information about this, let me know in the comments or on twitter.

Resources
Thinking in Java

Java Performance: The Definitive Guide: Getting the Most Out of Your Code

Optimizing Java: Practical techniques for improving JVM application performance