WebLogic Server and entropy

Peter Lorenzen
16/10-2013

After input from Jacco H. Landlust and Edwin Biemond I have rewritten my post about entropy.

Overview

Entropy pool

In computing you often need random numbers. They are used for encrypting stuff but also for lots of other things.

For Linux servers random numbers are default provided by the /dev/random device. /dev/random is a pool of random bits and it is called the entropy pool. The bits are calculated from user-triggered events (keystroke, disk I/O, mouse clicks etc).

/dev/random is a blocking device. When there is not enough random bits left it will block and the process that are reading from it will hang until more random bits has been generated.

Sometimes you will experience that it takes ages to start a Node Manager or a WebLogic Server, this will often be because of low entropy.

I often experience this since I install WebLogic on a lot of new servers. It happens both for virtual servers and physical servers.

In the past I have been scratching my head for 10+ minutes for the initial start of the Node Manager on a powerful Blade without any load.

Monitoring entropy

The default size of the entropy pool is 4096 bits. You can see the size here.

cat /proc/sys/kernel/random/poolsize 

You can see how much entropy is available via:

cat /proc/sys/kernel/random/entropy_avail

Or use watch if you want to display it every second:

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

If you are testing and want to drain the server for entropy you can use this script.

Using /dev/urandom

A way to solve the problem with /dev/random is to use the /dev/urandom device instead. It does not block when it runs out of user-triggered data, instead it reuses the bits it have left.

This of cause means that it is not as random as /dev/random and therefore not as secure.

I never use /dev/urandom on production servers, but how big a security issue urandom is I don’t know. If the entropy pool is not depleted often I assume that it is not a big issue.

Here is mentioned that /dev/urandom is not recommended for generation of long-term keys. So you should probably not use it for SSL private keys etc.

You can use /dev/urandom in different places. Here are 4 examples.

CONFIG_JVM_ARGS

You can use urandom when you configure a new domain.

For example:

export CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"
/u01/app/oracle/product/fmw/wlserver_12.1/common/bin/config.sh

Notice that you need to obfuscate the path to /dev/urandom a bit by using /dev/./urandom. Java has a “feature” where it will default to use /dev/random if you use /dev/urandom.

I don’t understand why this is but it has apparently been the case since 1.5.

setDomainEnv.sh

This will cover all the WebLogic Servers in the domain.

if [ "${USER_MEM_ARGS}" != "" ] ; then
        MEM_ARGS="${USER_MEM_ARGS}
        export MEM_ARGS
fi
MEM_ARGS="${MEM_ARGS} -Djava.security.egd=file:/dev/./urandom"

JRE – java.security

The JRE contains a file we you can configure where it should get its entropy from. This will cover all the WebLogic domains that use the JRE, but you will have to remember to change the file every time you install a new JRE version/patch.

$JAVA_HOME/jre/lib/security/java.security

Default /dev/urandom is configured, but as mentioned this is ignored by Java.

#securerandom.source=file:/dev/urandom
securerandom.source=file:/dev/./urandom

rngd daemon

You can use the rngd tool to feed random bits directly to the entropy pool. rngd is meant to be used to help out if you use a hardware random number generator, but can also help without one.

This will generate random bits via /dev/urandom and feed it to /dev/random.

rngd -r /dev/urandom -o /dev/random -b

It will keep adding bits to the entropy pool until the size is 2048 bits.

I use this if I have a production servers that is stuck because of low entropy.

rngd is from the rng-tools package and it has been installed by default on the Red Hat servers where I have used it.

You can also use rngd as service. It will continually monitor the entropy pool and add bits when needed.

You configure it by editing /etc/sysconfig/rngd.

EXTRAOPTIONS="-i -r /dev/urandom -o /dev/random -b"

This will every 60 seconds add bits to the entropy pool until the size is 2048. You can change the interval and size via the -t 60 and -W 2048 parameters. Notice that if you decrease the time or increase the size more of the entropy will come from /dev/urandom.

Add the service.

cd /etc/init.d
chkconfig --add rngd
chkconfig --level 35 rngd on
chkconfig --list rngd

Start it.

cd /etc/init.d
./rngd start

Additional information

“How to diagnose a Linux entropy issue on WebLogic Server instances? (Doc ID 1574979.1)”
“E1: OS: Linux Servers Hang or Have Delays on Any JAVA Process Affecting Performance (Doc ID 1525645.1)”
Security Analysis of Pseudo-Random Number Generators with Input: /dev/random is not Robust

{ 10 comments… read them below or add one }

jyothi October 25, 2013 at 05:40

Very good blog. Never ever heard about his topic and you have explained it so nicely that any one can understand it. Thank you very much.

Ingo Peters March 7, 2014 at 17:40

Excellent post that summarizes the solutions to this common problem in virtualized environments. I like the fact that you provide context as well as pros and cons to each approach.
And the cartoon is very appropriate!

Patrick March 12, 2014 at 10:45

Thanks for sharing, Peter. I am implementing it in our test environment right now.

sunil August 21, 2014 at 06:08

Hi Peter, How can we find that, we are hitting entropy pool shortage while utilizing random/urandom in our weblogic environment?

Peter Lorenzen August 21, 2014 at 11:07

As far as I know you cannot directly see this. If your WebLogic server takes much longer than normal to start and there is nothing unusual in the log files and /proc/sys/kernel/random/entropy_avail is low, you have probably depleted the pool. If you server hangs and you run rngd to add bits to the pool, and the server then promptly starts, it also indicates that there were a entropy problem.
Regards Peter

Pedro Morales August 27, 2014 at 21:26

Hi team,
You can take a thread dump and look at what ExecuteThread: ‘0’ or ExecuteThread: ‘1’ are doing. These two threads will run the deployment and other work to start the server. If in the thread dumps, you see the file IO classes, and the SecureRandomNumberGenerator or other stuff like that, then those threads are blocking for entropy.
Pedro

Peter Lorenzen August 28, 2014 at 11:04

Thanks Pedro

Arun September 22, 2014 at 23:05

You should be able to select the faster-but-slightly-less-secure /dev/urandom on Linux using:

-Djava.security.egd=file:/dev/urandom

However, this doesn’t work with Java 5 and later (Java Bug 6202721). The suggested work-around is to use:
-Djava.security.egd=file:/dev/./urandom
(note the extra ‘/./’)

From:
http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom

Torsten Kleiber October 1, 2014 at 06:40

Hi!

We tried to install rngd daemon, but when cannot start it, it say it need some hardware random generator.

Kind regards
Torsten Kleiber

Peter Lorenzen October 1, 2014 at 08:22

Hi Torsten,

I have only used it on Red Hat 6 and there it was already installed.

Regards Peter

Leave a Comment

{ 5 trackbacks }

Previous post:

Next post: