Screamernet

Introduction

The distributed rendering program that is packaged with LightWave is called ScreamerNet. It uses Layout’s Network Rendering Panel (Render > Network Render) to control submitting scenes to networked computers running the ScreamerNet process. ScreamerNet can control up to 1,000 CPUs. Each CPU takes a frame, renders it, and then grabs the next available frame in the animation until the scene is complete.

Are You Talking to Me?

Before ScreamerNet is discussed, your computers must be talking to each other, which means they need to be networked together. Setting up a network is beyond the scope of these instructions, though the experience can be quite rewarding.

Share and Share Alike

Once you have created your network and the computers are talking to each other, you’re ready for the next step. On our render farm, let’s say that we have four computers (Pig, Horse, Cow, and Chicken) networked together. (For now, imagine that they are all single CPU machines.)

In a ScreamerNet setup, you designate one of the computers as the control or host machine. It hands out the rendering assignments, (called jobs), to the nodes (the CPUs on the network used for rendering). The host must have LightWave installed on it.

In this scenario, Pig will be the control machine. Node machines do not need any NewTek software or hardware on them.

For ScreamerNet to work, all the nodes need to access specific files. Of course, all the nodes need to share the scene file and all its accompanying data including object and image files. If you’re organized, all these will be tucked away in your LightWave Content directory. Also, the nodes need to save the rendered frames to a shared directory. Each node must be able to load a plugin configuration file (LWEXT2018.CFG) that has the plugin files mapped with a path that the node can access. Finally, the ScreamerNet command program, (LWSN.EXE), needs to be visible to all the nodes.

Perhaps some of the confusion about setting up a ScreamerNet system arises because there is more than one way to organize a sharing scheme and you may have read tutorials that herald one method over another. If set up properly, all the schemes will work. So depending on your situation and disposition, you should choose the most suitable one for you.

Windows Setup

Drive Mapping

To share, data must be at a unique address that all the nodes on the network can access. Normally, your scene file is sitting in your LightWave Content directory, which is typically on your C drive. In our case, the scene, oldmacdonald.lws, would be in Pig, the host computer. Now in our animal farm network, Pig, as the host, would distribute out a rendering job that said, “Render oldmacdonald.lws that is on the C drive.” The problem is that when Horse gets that instruction, it looks on its C drive and can’t find the scene. So Horse says, “neigh.” It cannot render a scene that it can’t find.

The solution is to map a unique network drive (or drives) that contains all the necessary LightWave files. Normally, these files will be on the host computer. If that is your case, go to your host computer and from your Start menu Open your computer. You should see your drives and the Map Network Drive icon above.

In the panel, when you specify a drive letter, make sure that it is unused by any of the computers on your network. For example, Pig might only use five drive letters, A to E, so letters F to Z are available. But Chicken may have a dozen hard drives on its computer, which means it is using drive letters A to L. If you mapped a drive on Pig with the letter G, Chicken could still be confused. In our situation, we’ll pick S for the drive letter.

Next, you specify the folder that you want to connect to this drive letter. There are two schools of thought about the folder. The orderly and efficient approach makes a folder with only the necessary files in it and attaches it to the mapped drive letter. The “kitchen sink” method simply picks the hard drive LightWave is in and selects that as the folder. For example, on Pig, LightWave is on the C drive. In the Map Network Drive panel, you would click Browse… next to the Folder entry box. Under Microsoft Windows Network, go to Pig and highlight the C drive. Click OK. The Folder box would read \\Pig\C.

Now if you want to be more elegant about your mapped drive, you can just map your LightWave folder. The process is the same, right click on the Network Places icon, choose Map Network Drive… and specify a unique drive letter. This time when you browse for a folder to attach, select your LightWave folder.

Both of the above methods assume that all the necessary files are in your mapped drive. If you’re organized, then all of your objects and images will be in your Content folder. Back in the old days of the 20th Century when humongous hard drives were nine gigabytes and drive space was scarce, often you would store your rendered frames on another drive. If your scene accesses any data from other drives or stores frames elsewhere, then those folders must be accessible to the network nodes. That means that you will need to map more drives.

For another way of sharing folders, Matt Gorner, in an exhaustive, informative ScreamerNet tutorial available , shuns the drive mapping technique. In his method, (which you should read about directly), he simply sets up network sharing access of his LightWave folder. Matt goes into great detail about the whole process.

Since all the computers on your network need to share files with this mapped drive, you will need to map that drive on all your computers. After you have, you can test the connection by copying files into that drive from the node computers.

Organizing Configuration Files

Unfortunately, we are not finished with this job. But before we move on, let’s discuss alternatives for content directories. Often you will read about ScreamerNet setups that construct two file directories, one specifically for ScreamerNet and another for when you don’t use ScreamerNet. This is a great deal of extra trouble to go to, given that you’ll see no difference in running LightWave on your workstation under a setup for ScreamerNet, and when you decide to render across your network, you’ll be ready to go without having to relocate a set of files for the scene in question. Of course, if you prefer that approach, then Matt Gorner explains a nifty way of organizing a dual setup.

The next steps involve changing certain LightWave configuration files so that they are ScreamerNet-friendly. Before you do, you should get those files where you can see them. Start by creating a new folder in your LightWave directory called Config. Now, click on the My Computer icon and do a search for LWEXT2018.CFG, (the plugin configuration file), and move it to the Config folder. It will be in the %USERPROFILE%\.NewTek\LightWave\2018 folder if you just want to go directly there. Do the same for the LW2018.cfg file.

Next, you need to point LightWave to that Config folder when you start the program. On your Windows desktop or in your Start menu, right-click on the Layout icon and select Properties.

Depending on your setup, in the Target box, it will say something like:

C:\LightWave\bin\Layout.exe

This points to Layout’s program file.

If you have set up the Config folder, change this line to read:

C:\LightWave\bin\Layout.exe -cc:\LightWave\config

The first part is unchanged. The second part, ( -cc:\LightWave\config), tells LightWave where to look for the configuration files. The -c is the config cue. The rest (c:\LightWave\config) is simply the path.

Command Directory

When you run ScreamerNet, the host sends out Job files to the nodes and receives Ack (Acknowledgement) files back. You need to set up a Command directory to store these files. So, in your LightWave directory, create a new folder titled Command.

Now open up Layout on your host computer. Click on the Render Tab and under Utilities, click on Network Render. In the Network Rendering Panel, click on Command Directory and locate the newly created Command folder on your mapped drive. In our case, S:\LightWave\Command.A message pops up that says,

ScreamerNet must be reinitialized whenever the command directory is changed. Initialize now?

Click Yes. You can then close the Network Render Panel.

Next, click Edit > Set Content Directory. Choose the content directory on your mapped drive, in our case, S:\LightWave\Content.

Finally, you need to remap your plugins. And again, there are two methods. The search and replace technique involves opening the LWEXT2018.cfg file in a text editor. The lines would originally look like this:

	{ Entry
	  Class “AnimLoaderHandler”
	  Name “AVI(.avi)”
	 Module “C:\\lightwave\\plugins\\input-output\\avi.p”}
	 { Entry
	  Class “AnimLoaderHandler”
	  Name “DirectShow(.avi)”
	 Module “C:\\\\lightwave\\plugins\\utility\\dvview.p”}

You would search for C: and replace it with your mapped drive letter. In our case, it would be S:. Save your changes and you’re done. This method is great if you like to dig in there, get your hands dirty and see exactly what you are doing.

In Matt Gorner’s solution, you click on the Utilities Tab in Layout and select Edit Plugins. Clear all your existing plugins and in the panel, choose Scan Directory. Browse to your mapped drive and highlight the support\Plugins folder. Hit OK and an Add Plugins window will pop up. Now, if you open the LWEXT2018.cfg file in a text editor, you should see that all of the plugins have been remapped to the network drive.

To save the changes to the configuration files, you need to quit LightWave. When you start it up again, the new configurations will be loaded.

Setting up ScreamerNet

All right, now that you have all the necessary files configured and in folders that can be accessed by all the nodes, it is time to set up ScreamerNet itself. In our example, we have four computers in our ScreamerNet render farm. Pig is the host and will be handing out directions to the other three rendering nodes. To keep organized, you will assign a unique number to each node.

To initialise a node for ScreamerNet, you need to create a separate file that you will store and run on each node. It should be a text document called startlwsn_node.bat. Open it up in a text editor and set it to read:

	cd S:\LightWave\Programs
	LWSN.exe -2 -cS:\LightWave\Config S:\LightWave\Command\job1 S:\LightWave\Command\ack1

You can use this example as a guide. It consists of two lines (everything from LWSN.exe to ack1 is the second line). Edit it to fit your specific setup and save it with a new name. Each computer that is a rendering node needs this file on it. You can call it whatever you want, just remember to keep the .bat extension. Be creative - YouScreamIScream.bat or SpeedKills.bat.

The first line indicates where to find the lwsn.exe program. The second line begins by running the ScreamerNet program in –2 mode. (There are –3 and -5 modes that are explained later.) Simply, leave this part of the line as it is.

Next is a –c command, which indicates where to find the configuration files. If you have been following along, you will have set up a Config folder and moved your LW2018.cfg and LWEXT2018.cfg files there. If it is located elsewhere, change this path.

The next path tells ScreamerNet where to find the job commands. Again, if you have been following along, you have created a Command folder and changed the Command Directory path in Layout. The last path simply indicates where the acknowledgement (ack) files are located. The numbers after job and ack indicate the node that is running this batch file. Each node has a unique number. Usually, if you are using the host computer as a rendering node, you assign it number 1. So, its batch file would read job1 and ack1. In our case, Pig would be node 1. You then number the other nodes. For example, Horse would be 2, Cow would be 3, and Chicken would be 4.

Each of the computers on your network needs a copy of this node initialization batch file customized to it. For example Horse’s file would read:

	cd S:\LightWave\Programs
	LWSN.exe -2 -cS:\LightWave\Config S:\LightWave\Command\job2 S:\LightWave\Command\ack2

Cow’s would say:

	cd S:\LightWave\Programs
	LWSN.exe -2 -cS:\LightWave\Config S:\LightWave\Command\job3 S:\LightWave\Command\ack3

If any of the computers have more than one processor, or a processor with multiple cores, then you could use multiple batch files and set each computer to rendering multiple frames. Let’s say Chicken is a dual processor machine. You might create one file for each processor. The first might read:

	cd S:\LightWave\Programs
	LWSN.exe -2 -cS:\LightWave\Config S:\LightWave\Command\job4 S:\LightWave\Command\ack4

The other would say:

	cd S:\LightWave\Programs
	LWSN.exe -2 -cS:\LightWave\Config S:\LightWave\Command\job5 S:\LightWave\Command\ack5

Obviously, you would have to name the batch files differently, such as StartNode4.bat and StartNode5.bat.

The main issue with dividing up your computers in this way is that they may have multiple processors but only one stock of RAM. If your scene is light you can get benefit from making multiple batch files per computer, otherwise it’s probably better not to these days.

When you have all the batch files placed on the node computers, create a shortcut for them on the desktop of the host computer.

Show Time

You are about ready to make the magic happen. Go back to your host computer, open Layout and load the scene that you want to render over the network. Make sure that the Content directory is set to the mapped drive.

Under the Render Tab, open up the Render Globals Panel. Sometimes, you may not want to render all the frames in the scene. What you can do is set the Render First Frame and Render Last Frame fields to the desired range of frames. Make sure that you have Auto Frame Advance checked. Also, check Save RGB. Remember, ScreamerNet only renders individual frames. It will not render animation files. In the RGB Files field, indicate the network-mapped path of where you are saving files. For example, S:\LightWave\Frames.

Save the scene.

Once the scene(s) are saved, click on the Render Tab and Network Render. The number that you enter in the Maximum CPU Number field should be the highest number node that you will be using. For example, if we would be using just node 1 (Pig) and node 2 (Horse), you would enter 2. But let’s say that Horse wasn’t available but Cow was. Even though, you are still using only two nodes, you would enter 3 so that ScreamerNet would look for node number 3, which is Cow.

Now go to the node computers and click on the batch file icons that you created. A black MS-DOS window will open up and before you can start reading what it says, a continuous flow of repeating messages will probably begin. Depending on your exact setup, the message will read like:

Can’t open S:\LightWave\Command\job1

This will be normal until LightWave initializes ScreamerNet. Once you have started all your nodes, return to Layout and click on the Screamer Init button.

After a brief pause, you will receive a confirmation of the number of CPUs detected and the Screamer Init field will say Ready.

Now, if you look back at the node’s MS-DOS window, the message will say Init and then Wait. The nodes are waiting for their job commands.

In the Network Rendering Panel, click on Add Scene to List and find the scene or scenes. They will be displayed in the Scene List in the order they were selected. Notice that the List indicates the First and Last Frame to render, the frame advance Step, and the % of those frames that have been rendered.

If you added a scene by mistake, highlight it and click Remove Scene. If you want to remove all the scenes, click Clear List.

When the list is complete, click on Screamer Render. The job commands will be sent to the nodes, which are waiting in line. Node 1 receives the first frame, node 2 will take the second, and so on down the queue. If you look back at the node’s window, you will see it document its progress in the render.

As soon as a node finishes a frame, it receives the job to render the next available frame in the scene. When the scene is completed, ScreamerNet jumps to the next scene in the list.

When you are finished rendering all the scenes and are ready to close ScreamerNet, click the Screamer Shutdown button. A panel will pop up that says, “Are you sure you want to shut down the ScreamerNet CPUs?” If you choose Yes, the nodes are closed and their MS-DOS windows disappear. Now, if you wish to start a new session, you must restart ScreamerNet on each CPU and re-initialize the CPUs from the host machine.

If you want to stop rendering before ScreamerNet is finished, press the Esc key to abort the session. A message will appear in the Screamer Init window saying, “Waiting for CPUs to finish rendering.” If the scene is complex or one of your nodes is slow, waiting could take a quite a while.

Screamernet Time Flag

A new flag has been added to Screamernet with different operation in the main two modes. In -2 mode, the -t value indicates the polling time for the job file (how often it should be checked for new commands). In -3 mode, there’s no concept of a “job” file, so -t is used to indicate the frequency of diagnostic output during a render.

Installing Nodes

When installing Screamernet nodes for LightWave 2018, make sure that the Microsoft Visual C++ 2008 SP1 Redistributable Package is installed on each render machine running LWSN.exe. Normally, this package gets installed along with LightWave when using the LightWave installer. However, not everyone uses the installer when setting up LWSN.exe on a render node.

For a LightWave 2018 render machine running a 64-bit version of Windows, install the Microsoft Visual C++ 2008 SP1 Redistributable Package (x64). For a LightWave 2018 render machine running a 32-bit version of Windows, install the .

Rendering Without LightWave

The LWSN program has a third option that lets you render a scene without running LightWave. There is no control machine and thus it is not a distributed rendering situation. You must tell the program specifically what to render. The method is run from a DOS prompt using the following syntax (one line):

	LWSN -3 [-c<config dir>] [-d<content dir>] <scene file> <first frame> <last frame> [<frame step>]

As you can see, you supply the program with the basic information needed to render a scene. An example would be:

	LWSN -3 -cS:\Lightwave\Config -dS:\Lightwave\Content oldmacdonald.lws 1 900 1

In the example, the program would render frames 1 through 900 of the oldmacdonald.lws scene using the config files stored in the S:\Lightwave\Config and using S:\Lightwave\Content as the Content Directory.

Preparing Scenes for Screamernet

Since Screamernet is designed to function without interaction it is important that you prepare scenes completely before submitting them to your farm. This means making sure all your objects and images load correctly and that all your rendering and saving options are set before you go. Here’s a simple checklist:

1. Are you using LightWave’s content directory for your scene? If you are unsure, use the File > Package Scene tool to create a new content directory and populate it with the scene you currently have loaded.
2. Have you set your camera’s properties correctly? Do you want Depth of Field or Stereo? Now is the time to decide because once you send your scene to the farm, the only way to make that “Oh, I forgot such and such” change is to stop the farm and start it again. Make sure your anti-aliasing is sufficient.
3. Have you set your output options? Screamernet can’t save animations, so make sure that each frame’s images are being saved in a suitable format.
4. Are you using any third-party plugins? Make sure that your nodes have the plugins too, otherwise the renders will fail.
5. Have you locked your configs? To be sure they don’t get overwritten while rendering, making sure your config files are read-only. Also, turn off Autoscanning plugins in Options.
6. Have you baked? Things like flocking, dynamics and even deformations are best baked to MDD files to ensure they are exactly the same across your render nodes. It also means that those nodes only need to render final images and not mess with dynamics calculations.
7. Have you set a frame range? This one is last since it’s not something you will always do, but Screamernet renders whatever range has been set up in the scene. If you’d like some nodes to render the scene backwards and some forwards to meet in the middle, set a frame range for it and save a different scene file.

Troubleshooting and Limitations

The most common cause of ScreamerNet crashing is when too many nodes try to write or read their information to or from the host computer while the host renders. If that seems to be the case, try saving the frames to a mapped network drive on another computer. You will have to go back to your scene and redirect the path for saving frames. Of course, make sure that the new drive has enough space to store the frames.

If that doesn’t solve your crashing problems, do not use the host machine as a render node. Use it only as a server where the hard drives are found.

ScreamerNet can be finicky about how filenames and directories are composed. Putting spaces in a name, for example, old macdonald.lws, might cause a problem. Try using a dash, -, or an underscore, _, instead of spaces, old_macdonald.lws. You should also never use accented characters such as é, à or ç since there is no guarantee that these characters will all be rendered correctly through your farm.

Some plugins don’t like running with more than one thread. For ScreamerNet scenes, go to the Render Globals Panel, (Render > Options > Render Globals). In the Multithreading drop down menu, change to 1 Thread, and save the scene. Quit LightWave and start it again. This will update the configuration files. You then will need to restart all the nodes so they will read the updated configuration files.

Hopefully everything is running smoothly, but problems can occur, usually because something is setup wrong. Some of the main problems that arise are listed here along with possible remedies.

The biggest limitation to ScreamerNet are plugins that are not written to take advantage of it. I’m not aware of a definitive list of all the ones which do and don’t work. If in doubt read the documentation that came with the plugin, it should say whether it has problems or not. If it doesn’t say but the plugin still doesn’t seem to work, re-scan your plugins to update the LWEXT2018.cfg file. If it still doesn’t work then chances are it’s not compatible with ScreamerNet.

If your scene uses procedural textures (fractal noise etc.) you may experience differences in the pattern if you render on machines with different processors. Mixing old and new processors; AMD and Intel; could lead to problems. This is really only a problem with animations as textures could suddenly change from frame to frame, if in doubt run a test or use all computers with the same processor type.

• Problem - “My nodes can’t find the job files.”
• Possible solution - They can’t find the job files because they can’t see the Command folder, which holds them. So open up the node batch file and check the script lines that end in job and ack. Make sure that pathname before these words point to your Command folder.
  
  Also check that all the render node computers have access to read and write to the folder.
  
  
• Problem - “My nodes seem fine but when I press Screamer Init it can’t find any CPUs.”
• Possible solution - Again this is a command folder problem, check that the Network Render Panel has the same Command Directory path as the nodes batch file. Also check that the host computer has access to read and write to the folder.
  
  
• Problem - “Rendering seems to be working but no files are saved.”
• Possible solution - The messages output in the node’s MS-DOS window will often give you a clue to the problem. For example, it might say that it could not find an object or image that was in the scene. This can be caused when the Content directory that the rendering node is accessing does not contain all the scene’s data. Check that the render batch files are looking in the correct place for the config files, and that they can access the folder on the network.
  
  Next, make sure your scene file is saving RGB files and not an animation (.MOV / .AVI etc.) in the Output Tab of the Render Globals Panel. ScreamerNet can ONLY save individual frames.
  
  Finally, check that your plugins (LWEXT2018.cfg) file is up-to-date by re-scanning the plugins using the pathname all the nodes will be using to find them on the network.
  
  
• Problem - “Rendering seems to work okay but all my saved files are in .FLX format.”
• Possible solution - There are two possible reasons. The first is that ScreamerNet can’t find the plugin to save in the format you’ve specified, so check that your plugins (LWEXT2018.cfg) file is up to date by re-scanning the plugins using the pathname all the nodes will be using to find them on the network. Also check that the render batch files are looking in the correct place for the config files, and that they can access the folder on the network.
  
  The second reason is that the render node that saved the file ran out of memory to load the saver plugin so, as a last resort, it used the built-in FLX saver.
  
  If you have a whole scene’s worth of FLX images, you can use LightWave to convert them into your desired format. Simply load the series as a sequence of images for the Backdrop in a blank scene and render in the proper format. If you only need to convert a few FLX images, load them into the Image Editor Panel . Highlight the name of an image and double-click on the preview, (or just double-click on the name), to open it up in the Image Viewer . Now, click on File > Save RGBA , and pick your format.
  
  
• Problem - “My particles aren’t working.”
• Possible solution - Particles are a mathematical simulation. Therefore, each computer would start calculating the particles at the start of each frame rendered. The way to fix this is to “bake” the motion of the particles. You need to save the particle FX calculation to disk (as a .PFX file) in the folder with the scene file, or to a mapped drive that can be seen by all the networked computers.
  
  
• Problem - “Motion designer isn’t working.”
• Possible solution - Same as particles, this is a mathematical issue. The way to handle this is to bake the motion, and save the motion designer calculation to disk (as a .MDD file) in the folder with the scene file.