In a recent post to Not Possible IRL, Alpha Auer lamented the unfortunate experience she had at Hair Fair 2009. Stuck in molasses, she desperately struggled to move any distance at all. And the molasses wasn’t part of this year’s Candy Land theme; it was lag.

What caught my attention was that she tried to reduce the problem by switching off various Render Types in her client and, when that didn’t help, she assumed that the cause of the lag was all of the sculpties being used in the booth decorations this year.

Of course, her attempts to reduce her lag failed and the conclusion she drew about the lag of the lag was completely illogical conclusion, based on an all-too-common misunderstanding of the nature of lag. The vast majority of Second Life users seem to have very little understanding of what lag is and what causes it. I have decided to help to rectify this problem with my first Curio Obscura tutorial.

---

Lag 101: What is Lag and Why Does It Happen?

Everyone hates lag. Lag drives us crazy when our client gets so bogged down that our dance party turns into a series of still images. Lag frustrates us when we cannot move in the world or we start moving but cannot stop. Lag bores us when we have to wait for ten minutes just to see what is right in front of us. “Lag lag lag… All we do is frickin’ lag!”, we cry. Except we don’t say “frickin'”. We use a less lady-like verb.

Then we try to find ways to reduce the lag. We start pointing the finger at our video card, sculpties, scripts, HUDs, or, most frequently, Linden Lab. But what most people fail to realize is that what we are using the single term “lag” to describe three very different types of “lag”, each with their own causes and manifestations. The three forms of lag are Client Lag, Server Lag, and Transfer Lag.

The easiest way to see that these are different types of lag is to open your Statistics Bar by pressing Ctrl-Shift-1 or selecting View / Statistics Bar. The key measurements to examine here are FPS, Bandwidth, and Time Dilation. Each of these describes how efficiently a different part of the Second Life experience is running and each is affected by a separate form of lag.

---

Client Lag

Client Lag is lag which directly affects the Second Life client you have running on your computer. The more Client Lag you experience, the lower your FPS will be. FPS stands for “Frames Per Second”. It describes how many times your screen updates every second. Ideally, you want an FPS of 24 or better as this is the speed at which the illusion of movement becomes convincing. A lower framerate can still look like movement but becomes less and less convincing the lower it becomes. A higher framerate becomes much more convincing and can dramatically improve how realistic secondary motion feels, such as the gentle sway of a skirt or tail.

Personally, I run Second Life with a slightly older computer so I often operate at a framerate in the ten to twenty range. This is less pretty than I would like but certainly high enough to be functional. The framerate shown in the Statistics Bar above is higher than I can usually get with my old computer. I only got it that high by turning everything down and looking at the sky.

In extreme cases, when the FPS drops below 1, it can become almost impossible to interact at all since a lot can happen in between frames and yet you can only affect your avatar and the world once per frame. Suddenly, you cannot control your turning properly, you run into walls or off of a cliff because your client cannot keep up with what is happening. In essence, Client Lag makes you too slow for the world.

Client Lag has various causes. Other applications running on your computer can use of your computer’s resources, slowing down your client. Low quality hardware, such as an old motherboard or video card, keeps your client lagging behind like a horse and buggy at the Indy 500. Asking the client too render too much at once makes each frame take longer and longer to render.

The nice thing about Client Lag is, since it only affects each user individually, we have the most control over it. We can stop running other applications or keep them minimized. We can purchase newer hardware. Or we can turn down our Preferences in the Second Life client.

While reducing our Preferences will make what is on screen easier for the client to render, it still has to render everything it sees to some degree and, if there is a LOT to see, the framerate is only going to get so high. Large crowds of avatars wearing high-prim attachments can slow your framerate dramatically, a very unfortunate dilemma since many people love to wear their most flashy, attention-getting attachments when going to an event.

However, you still have some control over the effect of other people on your client. In extreme cases, you can choose to Mute another avatar, causing them to be rendered as a very simple cloud instead of the 20,000 prims they were wearing in a jewelled wristwatch. Of course, you won’t be able to hear them talk anymore so only use this approach if you don’t need to hear them talk.

You can also disable the rendering of all avatars through the Advanced menu but this will also prevent your avatar from rendering and your camera position won’t change when you move, making it very difficult to do anything in Second Life. Only use this solution in the most extreme cases, such as a large, seated event where all you really need to do is chat and you don’t care about being able to see anything.

Scenery containing too many polygons or too many high resolution textures can slow your framerate no matter what you do. If even the lowest graphics settings cannot help your framerate enough, the only solution for Client Lag caused by excessive scenery is to go somewhere else.

Sculpties use more polygons than a box but the same or fewer polygons than a sphere or torus. They do require an extra texture to download but, once downloaded, they often produce less Client Lag than regular prims since they are just one prim while making a similar shape out of regular prims would require many more prims and, therefore, many more polygons. In terms of Client Lag, sculpties tend to be less costly than regular prims.

It is important to keep framerate in mind when designing an environment in Second Life. You can dramatically improve framerate for your visitors by reducing the amount of objects and textures they can see at any one moment.

Try not to use very large textures on very small objects since the larger textures means larger amounts of data to send to the video card for each object. Even more important, don’t have too many different large textures visible all at once or the video card will be so busy copying texture data that it won’t be able to keep up. If you have a lot of vendors with large textures displayed, each one of those textures costs the video card a more memory. If you have more textures visible at once than can fit on the video card at one time, it has to swap memory back and forth with the motherboard to keep up with it all and that will slow down the framerate significantly.

You can also try to keep areas isolated from each other by adding more solid interior walls that block distant objects from being visible. This can reduce the number of textures that can be seen at one time inside a building and it can also reduce the amount of extraneous texture that can be seen outside. A home with walls made of huge glass windows may be pretty but it also means your client has to constantly render all of the neighboring homes as well.

Flexible prims are handled entirely on the client so the lag created by animating flexible prims only affects Client Lag. This is much better than trying to wiggle an object with prim animation but excessive use of flexible prims can reduce the framerate for everyone around you.

---

Server Lag

Server Lag is lag which affects the server on which the whole world of Second Life runs. This can apply to any of the servers involved such as the server for the region you are visiting, the server which handles your inventory, the servers for chatting with your friends, or the server for handling your monetary transactions.

You have absolutely no control over those last three servers. Only Linden Lab and their Internet Service Provider has any affect on the ability of those servers to keep up with demand. Thankfully, Linden Lab has a full-time, around-the-clock staff dedicated to monitoring and maintaining those servers so that, when problems do occur, they can fixed as quickly as possible. Certainly, we all wish that problems with the servers never happened. We also wish we had a pony. But life is what it is and servers aren’t made of magical adamantium. We simply have to trust that Linden Lab will always do their best to keep their customers happy. Given the sheer size of the system their are maintaining, Linden Lab does a remarkably good job at keeping our virtual world continuing to turn.

The one form of server lag that residents have any control over is lag on the server for each region. If you examine your Statistics Bar, you can see a section titled Simulator and a graph for Time Dilation. This is our best measurement for Server Lag. Time Dilation is the ratio of script time to real world time. The lower the value of Time Dilation, the slower scripts and physics can process on the server. That means that objects will start to interact slowly or erratically and avatars will find it hard to stop moving, or else to move at all, through the simulated physics of the region.

Ideally, you hope to see Time Dilation stay at or near 0.99. The effects of Server Lag are severe so even a small drop to 0.8 or 0.7 can have noticeable effects. If the average starts getting below 0.5, the region is in serious trouble.

The primary causes of Server Lag on the region are scripts and physics.

Every scripts uses up a little more of the server’s resources, whether it is a script in the scenery or in an attachment being worn by an avatar in the region. Some scripts do very little, updating very rarely or only once, and therefore contribute very little to lag. Other scripts run on timers or events that process many times a second, performing a large number of calculations, and contribute more heavily to the Server Lag in the region. Ultimately, it is not the size of the script but, rather, the number of events in the script, the amount of code in each event, and how often those events trigger. Efficient scripting is skill which can make a huge difference in how a server runs.

Complex objects or attachments that use a lot of sensors, listeners, or rezzing of other objects or excessively dynamic objects that use a lot of prim animation can contribute significantly to Server Lag on the region. Efficient design can help reduce the cost of scripts and animation on the server’s resources and make visiting the region a better experience for everyone. This applies to avatar attachments and HUDs as well. Don’t be a rude guest by lagging the server for everyone around you just to support a radar HUD or excessively waggy tail.

Physics is processed on the server so the more physical objects moving around the region, the heavier the burden on Server Lag. Every avatar is itself a physical object so the more people in the region, the more drain on the physics simulation. Additional physical objects, such as bullets, vehicles, and bouncy balls each cost the physical simulation a little more. They can be an even greater drain than an avatar since avatars don’t roll around but, at large popular events, the avatars will greatly outnumber other physical objects.

The amount and complexity of static, non-moving scenery also adds to the cost of the physics simulation but to a lesser degree than dynamic objects and avatars because static objects only have to be calculated once which dynamic objects have to be recalculated every time any of them move.

You can help reduce the physics cost of static objects in the scenery by setting unimportant objects to Phantom, thus removing them from the physics simulation, but a much greater effect can be had by reducing the amount of dynamic physical objects or reducing the number of avatars allowed in the region at one time.

There is a very key fact to understanding how scripts and physics affect the server. Physics and scripts have different priorities on the server. Scripts will only be processed when nothing else is happening. So scripts don’t interfere with the speed of physics, with your ability to move around the region, or with the speed with which you download data from the server. This does mean that scripts will slow down or behave sluggishly when there is a lot of activity in the physics simulation, either from dynamic objects or avatars, but it is much better than the alternative.

Even worse, every avatar in a region needs to know about every other avatar and download the textures for that avatar. That means that the server has to send data for every avatar to each avatar at a rate that grows geometrically. With fifty avatars in one sim, that means 2500 downloads in total just to handle all of the avatars.

Ultimately, this means that the more scripts there are, the slower all of the scripts will run but they won’t make the physics, movement, or downloading any slower.

---

Transfer Lag

Transfer Lag is lag which affects how quickly data is able to download from the server to your computer and upload from your computer to the server. This affects how quickly the prims around you appear, how quickly the textures on those prims go from blank grey to full detail, how quickly sculpties change from vague, round blobs to complex, graceful blobs, how soon an animation begins and how well synched are the animations on a couple, and how quickly a sound starts playing, and how smoothly prim-based animation performs.

In the Statistics Bar, transfer is measured in Bandwidth. The more Bandwidth, the more your client is downloading or uploading, the more Transfer Lag you will notice.

The more prims, textures, animations, and sounds there are in the region, the more there is to download. If those prims have scripts, they be change their position, textures, animations, and sounds periodically, creating even more data to download.

This is where sculpties seem to be more laggy than other prims because they one additional texture to download, the sculptmap, before they take shape. Since that sculptmap is often in lossless format, it can take longer to download than other textures. This is why sculpties are often the last prims to fully resolve when you enter a new area in Second Life. They require a more data to download than regular prims.

Of course, that assumes that there is only one texture on the regular prims instead of a different texture on every face of the prim. Since sculpties can only have one face, they only gave a maximum of two textures to download. By contrast, a box or tube can have up to ten textures to download.

Transfer Lag is tedious and annoying. It is very noticeable since it makes objects and avatars look grey and shapeless or blurry and unreadable. However, compared to the other two forms of lag, it is least important form of lag since it doesn’t prevent you from moving around the region, it doesn’t affect physics since that is handled on the server, and it doesn’t interfere with scripts since those are all processed on the server. All Transfer Lag does it annoy us and make us impatient but it is ultimately self-correcting if we simply wait long enough. However, because Transfer Lag is so visibly obvious, we often mistake it for being related to other forms of lag even when it has nothing to do with those issues.

One other cause of Transfer Lag is your connection to the Internet. If your ISP is having problems or your modem is slow or failing, you could experience a lot of Transfer Lag. In addition, you might even experience Transfer Loss where some data will never ever reach you at all. This has nothing to do with anything in Second Life and can only be repaired by replacing your modem or having your ISP fix the problem.

---

Now that we have identified the three forms of Lag, let’s consider some specific examples.

Particle Effects are performed entirely on the client. Excessive particles can cause Client Lag but you can reduce the maximum number of particles to render in your Preferences. Particles are much less expensive to render than prims. Particles only cause Transfer and Server Lag if they are changing parameters very, very frequently, as with a complex fireworks show.

Omega rotation is a special case. If the object is physical, omega rotation can be very expensive in both Server Lag, as the rotating object has to interact with the physics simulation, and Transfer Lag, as the object has to constantly update its position with the results of the rotation and physics. However, if the object is not physical, the rotation is handled entirely on the client at a very low cost, and the physics simulation simply treats the object as if it were not rotating at all. Keep in mind that, in this case, being physical specifically refers to having the Physical checkbox checked. It does not need to be Phantom in order to be rotated by the client without physics.

Sculpties require an extra texture download before they become visible. This is Transfer Lag. It only affects how quickly we can see the object but it has no effect on framerate or server performance. They are often more efficient than regular prims since it requires many more regular prims to make a similar shape and each additional prim adds to the physics simulation, causing more Server Lag, and may have more textures and polygons, adding to the Client Lag.

Scripted vendors cost the server additional resources and contribute to Server Lag. However, with Mono-compiled scripts, each script costs much less than it used to cost. In addition, most vendor scripts are efficiently designed so that they have events triggering when someone is directly interacting with them. Unless they are running a slideshow or rezzing holographic samples, most scripted vendors are relatively low cost to the server.

Animation Override systems can contribute a lot to Server Lag since they have to constantly monitor the animations that the avatar wants to play and decide which custom animation to play instead. In addition, they create Transfer Lag by adding more animations that each client needs to download. For these reasons, it is best to remove or disable Animation Override systems when entering a heavily populated area.

Hair contains a lot of prims, including a lot of textures containing transparency. However, most hair uses the same couple of textures over and over, dramatically reducing the render cost. And, while it may be a large number of prims, it is one attachment. This is also an example of where sculptie efficiency comes into play as a hairstyle made with sculpties will usually have only a fraction of the number of prims in a hairstyle without sculpties. Overall, while the size and complexity of a hairstyles does add to the Client Lag simply by being one more thing to render, and it does add a little more to the Transfer Lag by adding a few more textures and objects to download (as does any attachment), it has no real effect on Server Lag.

Resize scripts can come in different forms. Some approaches to a Resize script require one parent script in the object and then a second smaller script in every child prim. In an object with 100 prims, that means at least 100 scripts, each child script listening for link messages from the parent script. By contrast, some approaches require only one script in the object. That script tends to be larger but it means a lot fewer events to process and trigger. Inefficient resize scripts can contribute to Server Lag but mostly only when they are actually being resized. The rest of the time, the extra events do cost a little bit of extra server processing, just to verify that they haven’t triggered, but not as much processing as when they do trigger.

Combat HUDs, Radar HUDs, and other tool-based HUDs can produce a LOT of Server Lag. They use frequent sensors to learn about other objects in the region, they may have listeners which are triggering on everything everyone says, and they may even be rezzing additional objects that spread themselves around the region with their own sensors and listeners. Apart from griefers deliberately trying to crash a server, these can be the most significant causes of Server Lag.

---

Let’s consider the situation at Hair Fair 2009. If you have managed to visit the Hair Fair, you will have observed many symptoms of lag. It takes a long time for many of the booths and vendors to fully appear and for their textures to resolve and people can barely move, running in place without getting anywhere. All of which are very common problems with large, popular events.

As we can see, these are actually two completely separate forms of lag: Transfer Lag and Server Lag.

We can even verify which type of lag is manifesting by looking at the Statistics Bar.

I teleported into the Hair Fair, waited a few minutes, then checked the Statistics Bar.

The FPS is about average for my computer. Maybe a little bit below average but I can offset that by reducing my Preferences. So Client Lag is not much of an issue.

Bandwidth is high but that is to be expected since there are hundreds of vendors and textures all around me, in addition to all of the people around me.

Time Dilation is all over the place with a median value of 0.60 and even drops far below 0.5 sometimes. This is a terrible result for Time Dilation and indicates that Server Lag is very high. No wonder people are having difficulty moving around the fair!

Let’s consider the types of lag and possible causes at the Hair Fair.

The extensive use of sculpties in the Hair Fair decorations do add a little bit to Transfer Lag but, since the same sculpties are used all over the Hair Fair, the cost is very minimal.

More importantly, the cost of downloading a dozen sculptmaps is completely dwarfed by the transfer cost of downloading hundreds of different vendor textures. Since you certainly couldn’t have a Hair Fair without showing pictures of the hair on the vendors, there’s really nothing we can do but try to be patient.

The real problem at the Hair Fair is the difficulty people are experiencing moving around the Hair Fair regions. This is a form of Server Lag. We know the causes of Server Lag so we can diagnose a likely cause.

Server Lag is caused by scripts, physics, and attachments. We know that the only scripts allowed in the Hair Fair booths are the charity vendor scripts which are very simple, low cost scripts.

So the primary cause of lag at the Hair Fair can only be one thing: All of the people visiting the Hair Fair.

As we know, even if people are arriving at the event wearing lots of scripted attachments with full Animation Overrides and Sensor HUDs, this will not affect the physics simulation. It would only slow down other scripts. So if the problem we are seeing is avatars finding it difficult to move, the cause of the lag must that there are so many avatars trying to walk, run, and fly all over the region, creating a lot of extra drain on the physics simulation.

The problem is simply the size of the crowds. Since this is the fourth Hair Fair and each Hair Fair has been more popular than the last, we are currently seeing the Hair Fair regions staying at capacity much longer than ever before. There are also four regions for the Hair Fair, twice the number there were in 2007, and the server’s have to tell you about all of the people moving around within view on the neighboring sims as well.

These are the actual causes of the severe lag at the Hair Fair. Anyone who tries to blame the cause of the lag at the Hair Fair on the use of few dozen sculpties may be well-meaning but is misinformed.

---

For more information on lag and ettiquette, check out Anatomy of Lag by Gwyneth Llewelyn as a guest writer on Ana Lutetia’s blog.

You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

17 Responses: