DRQueue & Blender: Render Farm

February 3, 2012 Update: Heavy duty Work In Progress! I have filled in most of the commands and code needed, but I plan on elaborating with pictures and a diagram of my infrastructure. Also, installation on Ubuntu 11.10 has shown some special nuances that I haven’t added yet. Stay tuned!

Spend enough time on the internet researching render farms and it is inevitable that you will come upon the open source distributed render farm manager, DRQueue. Sounds great! Works with a variety of platforms and renderers, including Maya and Blender. It won’t take you long, however, to find yourself drowning in a sea of confusion and wires.

Contrary to popular belief, DRQueue doesn’t have to be a complicated setup. We’ll go so far here as to set up a render farm with DRQueue and get it on track with Blender 2.61, r43385+. If you don’t know what those numbers are, do not worry we’ll get you up to speed on the power and flexibility of versioning as well.

The concept of a RenderFarm is an interesting implementation of the Client/Server Model. In the RenderFarm scenario, the Clients (here referred to as “slaves”) do the bulk of the heavy lifting and the Server (here referred to as the “Master”) does some lightweight delegation of the renders to the client via another specialized Client known as the Queue Manager (here referred to as “DrQMan”).

Disclaimer: Perform anything mentioned in this blog at your own risk.

GOAL:

I will be building a Render Farm with a single Master Server and 10 Slave Clients. We are going to assume that they have already been wired up to the same 172.16.0.0/16 network. The Master Server has the name “rfserver” and each of the slaves are named “render###” with the number representing which slave they are.

We will be approaching this is 7 Steps.

1. Prerequisites: Libraries and Environment Variables

OS:

Ubuntu 11.04+ on i686 or x86_64 architecture.

CREATE ROOT:

sudo mkdir -p /mnt/que/python2.7.2/src 
sudo mkdir -p /mnt/que/python3.2.2/src
sudo mkdir -p /mnt/que/blender-svn
sudo chown username -R /mnt
chmod 777 -R /mnt

LIBRARIES & PROGRAMS:

sudo apt-get install vim subversion git-core build-essential g++ gcc pkg-config libgtk2.0-dev libglib2.0-dev libpango1.0-dev gettext libxi-dev libsndfile1-dev libpng12-dev libfftw3-dev libopenexr-dev libopenjpeg-dev libopenal-dev libalut-dev libvorbis-dev libglu1-mesa-dev libsdl1.2-dev libfreetype6-dev libtiff4-dev libavdevice-dev libavformat-dev libavutil-dev libavcodec-dev libjack-dev libswscale-dev libx264-dev libmp3lame-dev python3.2-dev scons libspnav-dev zlib1g-dev libncurses5-dev tcl8.5 tk8.5 swig csh tcsh fping samba smbfs samba-common system-config-samba nfs-kernel-server openssh-client openssh-server libncursesw5-dev libreadline-gplv2-dev libssl-dev libgdbm-dev libc6-dev libsqlite3-dev tk-dev gedit 

ENVIRONMENT VARIABLES:

sudo gedit /etc/environment

-Add to PATH, before close quotation mark:

:/mnt/que/blender-svn/blender:/mnt/que/drqueue/bin

-Add under PATH, the following lines:

export DRQUEUE_ROOT=/mnt/que/drqueue 
export DRQUEUE_TMP=/mnt/que/drqueue/tmp 
export DRQUEUE_OS=LINUX 
export DRQUEUE_MASTER=rfserver 
export DRQUEUE_BIN=/mnt/que/drqueue/bin 
export DRQUEUE_ETC=/mnt/que/drqueue/etc 
export DRQUEUE_DB=/mnt/que/drqueue/db 
export DRQUEUE_LOGS=/mnt/que/drqueue/logs 
export DISPLAY=:0

-initiate Environment Variables

source /etc/environment

Here’s what your /etc/environment file should look like, in theory:

Drqueue Blender Environment Variables

HOSTS SETUP:

We’ll set up the hostnames to their respective IP addresses in the “hosts” file. We will allow our current hostname to remain connected to it’s loop-back address, 127.x.x.x.

The “hosts” file is a part of the Name Resolution system and is usually called before the DNS.

“rfserver” is the hostname of the server, or “master”, of the renderfarm. We’ll make reference to it often.

The following example shows the “hosts” file of a single master Server and ten slave Clients.

/etc/hosts (ex: for the render001 machine):

127.0.0.1 localhost
127.0.1.1 RENDERFARM-render001
172.16.0.2 rfserver
172.16.0.3 render002
172.16.0.4 render003
172.16.0.5 render004
172.16.0.6 render005
172.16.0.7 render006
172.16.0.8 render007
172.16.0.9 render008
172.16.0.10 render009
172.16.0.11 render010

Here’s what mine looks like for the render005 machine:

Drqueue Blender Linux Ubuntu Render Farm

Yes, it would’ve been easier to put “rfserver” on a later ip address so that I could keep a consistent numbering scheme, but whatever…

GENERAL HOUSEKEEPING:

At this point we need to just make a few adjustments to make the installations run:

sudo cp /usr/lib/libtk8.5.so.0 /usr/lib/libtk8.5.so
sudo cp /usr/lib/libtcl8.5.so.0 /usr/lib/libtcl8.5.so

2. Render Farm Node Management: OpenSSH

OpenSSH provides us a means of ssh’ing into the Slave Client machines to finish the installations, run the slave and update the slave as needed.

CREATE PRIVATE & PUBLIC KEYS ON MASTER SERVER:

cd ~/.ssh 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render001 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render002 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render003 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render004 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render005 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render006 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render007 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render008 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render009 
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa.render010 
ssh-copy-id -i ./id_rsa.render001 render001@render001 
ssh-copy-id -i ./id_rsa.render002 render002@render002 
ssh-copy-id -i ./id_rsa.render003 render003@render003 
ssh-copy-id -i ./id_rsa.render004 render004@render004 
ssh-copy-id -i ./id_rsa.render005 render005@render005 
ssh-copy-id -i ./id_rsa.render006 render006@render006 
ssh-copy-id -i ./id_rsa.render007 render007@render007 
ssh-copy-id -i ./id_rsa.render008 render008@render008 
ssh-copy-id -i ./id_rsa.render009 render009@render009 
ssh-copy-id -i ./id_rsa.render010 render010@render010 

CREATE SSH CONFIG ON MASTER SERVER:

OpenSSH will use the .ssh/config file to make private key management much easier. Note: the “echo” line below is all one line.

touch ~/.ssh/config 
chmod 600 ~/.ssh/config
echo -e "Host render001\n User render001\n IdentityFile ~/.ssh/id_rsa.render001\nHost render002\n User render002\n IdentityFile ~/.ssh/id_rsa.render002\n\nHost render003\n User render003\n IdentityFile ~/.ssh/id_rsa.render003\n\nHost render004\n User render004\n IdentityFile ~/.ssh/id_rsa.render004\n\nHost render005\n User render005\n IdentityFile ~/.ssh/id_rsa.render005\n\nHost render006\n User render006\n IdentityFile ~/.ssh/id_rsa.render006\n\nHost render007\n User render007\n IdentityFile ~/.ssh/id_rsa.render007\n\nHost render008\n User render008\n IdentityFile ~/.ssh/id_rsa.render008\n\nHost render009\n User render009\n IdentityFile ~/.ssh/id_rsa.render009\n\nHost render010\n User render010\n IdentityFile ~/.ssh/id_rsa.render010\n" >> ~/.ssh/config

SET PERMISSIONS ON SLAVES & MASTER:

Biggest thing: do these steps in order!

chmod 700 ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa*
chmod 644 ~/.ssh/*.pub
chmod 644 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/known_hosts
sudo restart ssh

CONFIGURE SLAVE CLIENTS:

In /etc/ssh/sshd_config:

—-Make sure they are uncommented (aren’t proceeded by a #):

RSAAuthentication yes
PubkeyAuthentication yes
Banner /etc/issue.net

In /etc/issue.net you can set a person banner message everytime you log in. OR, you can just modify /etc/motd, which is the post-message: Message Of The Day. This is mine:

Utah Teapot Drqueue Blender Render Farm MOTD

LOG IN TO THE SSH SERVER OF A SLAVE CLIENT:

ssh render####

3. Render Farm Python Language Setup

We’ll be setting up necessary versions of Python here. We’ll need an installation of Python 3.2.2 and Python 2.7.2. From here, we’ll create the alternate versions accessible via /usr/bin/python and will show you how to switch between them, depending on our needs.

INSTALL ALTERNATE PYTHON VERSION 3.2.2:

cd /mnt/que/python3.2.2/src 
wget http://www.python.org/ftp/python/3.2.2/Python-3.2.2.tgz && tar -xvf Python-3.2.2.tgz 
cd ./Python-3.2.2 
./configure --prefix="/mnt/que/python3.2.2"
make
sudo make altinstall

INSTALL ALTERNATE PYTHON VERSION 2.7:

cd /mnt/que/python2.7.2/src
wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tgz && tar -xvf Python-2.7.2.tgz
cd ./Python-2.7.2
./configure --prefix="/mnt/que/python2.7.2"
make
sudo make altinstall

SETUP PYTHON3.2.2 AS ALTERNATIVE:

sudo update-alternatives --install /usr/bin/python python /mnt/que/python3.2.2/bin/python3.2 1 

SETUP PYTHON2.7.2 AS ALTERNATIVE:

sudo update-alternatives --install /usr/bin/python python /mnt/que/python2.7.2/bin/python2.7 1 

SWITCH ALTERNATIVES BETWEEN 2.7.2 & 3.2.2:

sudo update-alternatives --config python

4. Render Farm Queue Manager: DRQueue

BUILD & INSTALL DRQUEUE:

You will need to have the Python2.7.2 alternate chosen under “update-alternatives”, as described in the previous section.

cd /mnt/que/ 
git clone https://ssl.drqueue.org/git/drqueue.git 
cd ./drqueue/src
scons 
sudo scons PREFIX=/mnt/que install

DRQUEUE CONFIGURATION:

Now, we will configure Drqueue to work with our created system. We already set the proper environment variables back in section 1!

-Master [/mnt/que/drqueue/etc/master.config]:

logs=/mnt/que/drqueue/logs
tmp=/mnt/que/drqueue/tmp
db=/mnt/que/drqueue/db
bin=/mnt/que/drqueue/bin
etc=/mnt/que/drqueue/etc

master.config Drqueue Blender Render Farm Renderfarm

-Slave [/mnt/que/drqueue/etc/slave.config]:

The “pools” here will be discussed in the last section, where we will learn how to create Jobs & Tasks. For now, leave them as “test1,test2,test3″.

logs=/mnt/que/drqueue/logs
tmp=/mnt/que/drqueue/tmp
pool=yourpool1, yourpool2, ...

slave.conf Drqueue Blender Render Farm Renderfarm

-Queue Manager (“Client”) [/mnt/que/drqueue/etc/drqman.config]:

And now we configure our Que Manager, often referred to as the Render Farm Client.

logs=/mnt/que/drqueue/logs
tmp=/mnt/que/drqueue/tmp
db=/mnt/que/drqueue/db

drqman.conf Drqueue Blender Render Farm Renderfarm

5. Render Farm Renderer: Blender 2.61

By far, this step takes the longest. Blender will be the largest installation in this compilation, so I advise using multiple threads while installing with scons. We want to stay up to date on the latest version of Blender, so we will use Blender’s svn versioning to grab the latest and, hopefully, best of blender. Pay attention to what architecture you are using (64- or 32- bit). The lscpu command can help you out.

CHECKOUT BLENDER 2.61:

cd /mnt/que/blender-svn
svn co https://svn.blender.org/svnroot/bf-blender/trunk/blender
svn co https://svn.blender.org/svnroot/bf-blender/trunk/lib/linux64 lib/linux64 (FOR 64-bit machines)
svn co https://svn.blender.org/svnroot/bf-blender/trunk/lib/linux lib/linux  (FOR 32-bit machines)
cp ./blender/build_files/scons/config/linux-config.py ./blender/user-config.py 

CONFIGURE BLENDER 2.61 FOR BUILD & INSTALL:

vim ./blender/user-config.py

—In “user-config.py” file in blender-svn/blender, replace the following:

BF_PYTHON = '/usr/local
BF_PYTHON_LIBPATH = '${BF_PYTHON}/lib'
BF_PYTHON_VERSION = '3.2'
WITH_BF_STATICPYTHON = False
BF_PYTHON_INC = '${BF_PYTHON}/include/python${BF_PYTHON_VERSION}'
BF_PYTHON_BINARY = '${BF_PYTHON}/bin/python${BF_PYTHON_VERSION}'
BF_PYTHON_LIB = 'python${BF_PYTHON_VERSION}' #BF_PYTHON+'/lib/python'+BF_PYTHON_VERSION+'/#config/libpython'+BF_PYTHON_VERSION+'.a'
BF_PYTHON_LINKFLAGS = ['-Xlinker', '-export-dynamic']
BF_PYTHON_LIB_STATIC = '${BF_PYTHON}/lib/libpython${BF_PYTHON_VERSION}.a'

—with this:

BF_PYTHON_ABI_FLAGS_SUFFIX = "m" # may be any combination of 'dmu' or empty
BF_PYTHON = '/mnt/que/python3.2.2'
BF_PYTHON_LIBPATH = '${BF_PYTHON}/lib'
BF_PYTHON_VERSION = '3.2'
BF_PYTHON_INC = '${BF_PYTHON}/include/python${BF_PYTHON_VERSION}' + BF_PYTHON_ABI_FLAGS_SUFFIX
BF_PYTHON_BINARY = '${BF_PYTHON}/bin/python${BF_PYTHON_VERSION}'
BF_PYTHON_LIB = 'python${BF_PYTHON_VERSION}' + BF_PYTHON_ABI_FLAGS_SUFFIX
BF_PYTHON_LINKFLAGS = ['-Xlinker', '-export-dynamic', '-ltk8.5', '-ltcl8.5', '-lz']
BF_PYTHON_LIB_STATIC = '${BF_PYTHON}/lib/libpython${BF_PYTHON_VERSION}' + BF_PYTHON_ABI_FLAGS_SUFFIX + '.a'

—You also need to replace the following:

BF_FFMPEG = LIBDIR + '/ffmpeg'
if os.path.exists(LCGDIR + '/ffmpeg'):

—With (obviously “linux64″ would be “linux” on 32-bit machines):

BF_FFMPEG = '/mnt/que/blender-svn/lib/linux64/ffmpeg'
if os.path.exists('/mnt/que/blender-svn/lib/linux64/ffmpeg'):

BUILD & INSTALL BLENDER 2.61:

The “-j 3″ signifies how many threads to run the scons install. In this case, three. If you’re not sure how many cores/threads you have, use the lscpu command.

cd /mnt/que/blender-svn/blender
sudo python scons/scons.py -j 3
ln -s ../install/linux/blender ./blender

6. Render Farm File Server & Client: Samba & smbfs

MASTER SERVER:

Let’s collect some information and configure our server. The information we get here will also be used in the Slaves’ Client configurations.

serverusername is the user that you are creating the Samba Server on.

SERVER UID & GID ####:

id -u serverusername
id -g serverusername

SAMBA SERVER CONFIGURATION:

Go into the /etc/samba/smb.conf file:

sudo vim /etc/samba/smb.conf

Change/create the following lines:

-Under “global” section:

workgroup = RENDERFARM
 server string = %h RenderFarm DrQueue Server

-Under “Authentication” section:

security = user
usernamemap = /etc/samba/smbusers
encrypt passwords = true
map to guest = bad user

-Now, at the bottom, add the following Samba Share:

[masterrenderfarm]
path = /mnt/que
comment = DRQUEUE Master Share
valid users = serverusername
guest account = nobody
browsable = no
writable = yes
printable = no
create mask = 777
directory mask = 777
read only = no

CREATE SAMBA SERVER USERNAME & PASSWORD:

The serverusername that we use below will be the user that you use on the master machine to create the Samba Server. NOTE: It must be a user that already exists on the machine. I recommend it be the same user we are using to set this up on the Master Server machine. The serverpassword will be its password. The serverpassword does not have to be the same as the machine user’s password.

sudo smbpasswd -a serverusername
sudo smbpasswd -e serverusername 

Now create/edit the /etc/samba/smbusers file and add the following line:

serverusername = "serverusername"

Restart the Samba Server:

sudo restart smbd
sudo restart nmbd

SLAVE CLIENTS:

Fill in the bold section with your appropriate information. Take Note: The information pertains to the Server/Master machine. So the username and password are the same username and password of the user which the Master Server is on.

Create Credentials File for Client:

~/.smbcredentials

Add the following lines to /etc/fstab:

//rfserver/masterrenderfarm/drqueue/tmp /mnt/que/drqueue/tmp smbfs username=serverusername,password=serveruserpassword,uid=####,gid=#### 0 0

//rfserver/masterrenderfarm/drqueue/logs /mnt/que/drqueue/logs smbfs username=serverusername,password=serveruserpassword,uid=####,gid=#### 0 0

Now mount the Drqueue Shares. The UID & GID we got in the Server section:

sudo mount -t smbfs //rfserver/masterrenderfarm/drqueue/logs /mnt/que/drqueue/logs -o username=serverusername,password=serveruserpassword,uid=####,gid=####
sudo mount -t smbfs //rfserver/masterrenderfarm/drqueue/tmp /mnt/que/drqueue/tmp -o username=serverusername,password=serveruserpassword,uid=####,gid=####

Housekeeping: prevent hanging:

sudo update-rc.d -f umountnfs.sh remove
sudo update-rc.d umountnfs.sh stop 15 0 6 .

If you need to unmount a share from a folder, use umount on the folder upon which the share is mounted. You may need to “sudo” in.

7. Render Farming

A lot more info to come!

Here I’ll discuss actual render jobs and how to go about a typical Blender rendering with DRQueue. The “&” after the commands signify to run the process in the background so that you can continue to work in the terminal. Press enter after the proc header comes up and you’ll be back at the prompt.

architecture = i686 or x86_64

Start Master: master.Linux.architecture -l3 -o &

Start Slave: slave.Linux.architecture -l3 -o &

Start Manager Client: drqman.Linux.architecture -l3 -o &

End Clients: killall -9 slave.Linux.architecture

End Master: killall -9 master.Linux.architecture

End Manager Client: killall -9 drqman.Linux.architecture

TROUBLESHOOTING

If you can get on the right track of thinking when it comes to designing, building, configuring and executing a renderfarm before you ever type “sudo apt-get install…“, then you might can save yourself a good deal of grief.

So, why do things break down when it comes to setting up our render farms? 5 W’s time:

1. What?

To build a program, the source code for that program needs all of the necessary libraries (external code used at compile/run-time). We’ll try to grab all of these in the first section, Prerequisites.

2. Where?

The source code and built programs need their libraries and configuration files to work. If it cannot find them, then the system fails or produces unexpected results. We combat this through careful organization of directory structures and file paths.

3. Who?

Permissions, in my experience, are often the number one issue for properly compiled programs. We’ll cover our bases on making sure that we have full access when setting up our render farm. The trade-off with permissions is security. I won’t be covering security here.

4. When?

Programs and their libraries are often upgraded rather frequently. Sometimes

5. How?

About these ads

About agentjj20

Shua Jackson works as a VFX Supervisor out of the Upstate, SC. He also keeps busily involved in film directing and producing.

Posted on January 14, 2012, in DRQueue. Bookmark the permalink. 3 Comments.

  1. first of all thanks for the great detailed tutorial, i am running it on Ubuntu 11.04 and i reached the step where i have to copy the files as u mentioned using the command below:

    sudo cp /usr/lib/libtk8.5.so.0 /usr/lib/libtk8.5.so

    unfortunately i get the following error:

    cp: cannot stat `/usr/lib/libtk8.5.so.0′: No such file or directory

    do i have to get the mentioned library manually ? fetch it from somewhere ? or there is something wrong i did in the previous steps or is it just the in-compatibility of 10.04 version ?

    thanks in advance

  2. Hey there agentjj20

    at the step of compilation after executing the following command:

    sudo python scons/scons.py -j 3D

    it terminates and gives the following:

    Compiling ==> ‘AUD_PyAPI.cpp’
    scons: `/mnt/que/blender-svn/build/linux/lib/libge_phys_bullet.a’ is up to date.
    In file included from intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp:41:0:
    intern/ffmpeg/ffmpeg_compat.h: In function ‘int64_t av_get_pts_from_frame(AVFormatContext*, AVFrame*)’:
    intern/ffmpeg/ffmpeg_compat.h:149:1: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
    intern/ffmpeg/ffmpeg_compat.h:152:1: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp: In constructor ‘AUD_FFMPEGWriter::AUD_FFMPEGWriter(std::string, AUD_DeviceSpecs, AUD_Container, AUD_Codec, unsigned int)’:
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp:58:89: error: ‘avformat_alloc_output_context2’ was not declared in this scope
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp:167:7: warning: ‘int avcodec_open(AVCodecContext*, AVCodec*)’ is deprecated (declared at /usr/include/libavcodec/avcodec.h:3650) [-Wdeprecated-declarations]
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp:167:37: warning: ‘int avcodec_open(AVCodecContext*, AVCodec*)’ is deprecated (declared at /usr/include/libavcodec/avcodec.h:3650) [-Wdeprecated-declarations]
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp: In member function ‘void AUD_FFMPEGWriter::encode(sample_t*)’:
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp:252:1: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp: In member function ‘virtual void AUD_FFMPEGWriter::write(unsigned int, sample_t*)’:
    intern/audaspace/ffmpeg/AUD_FFMPEGWriter.cpp:292:99: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
    Compiling ==> ‘STR_String.cpp’
    scons: *** [/mnt/que/blender-svn/build/linux/intern/audaspace/ffmpeg/AUD_FFMPEGWriter.o] Error 1
    scons: building terminated because of errors.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: