Working with Raycasts and Layers in Unity

Unity allows for the placement of gameobjects into layers that can be used to mask items in and out of visibility with respect to the lights, camera or raycasts.  To examine these in action I have created a scene with a cylinder, cube and sphere thus:

Go ahead and create your own scene that looks like this in a new 3D Unity project.  Note there’s a smaller long cube (rotate 180 on the Y axis so it’s facing the sphere) at one end of the scene on the right that will act as the location of a raycast later.

To create a layer select Edit > Project Settings > Tags & Layers.  Then in the Inspector add the names for the new layer or layers. Notice several default layers (0 to 8 – layer 8 is added automatically by my version of Unity 2018, you might not have it and that’s okay) already exist.  Do not delete these.

Create three new layers called SphereLayer, CubeLayer and CylinderLayer.  Make note of the layers each of them are on.  For me SphereLayer begins at 9.

Assign each of the new objects you’ve created (the sphere, cube and cylinder) to their own layer by selecting the game object you want from the Hierarchy and in the Inspector assign it to its layer thus:

The simplest way to see layers in action is to turn them on and off on the camera. Let’s say you want to remove the sphere from view in the Main Camera. To do that, select the main camera and then set it’s Culling Mask to untick the SphereLayer. Anything in the scene assigned to the SphereLayer will disappear.  Note: you can add more than one item to a layer.

Culling objects from the camera view is quite handy when you have more than one camera or simply like something to disappear. You can also set the Culling Mask on a light to have it light only one part of a scene.  For example, you might want a character to have a blob shadow rather than real time lighting in which case you might not want the lights in the scene to illuminate the character.

Layers come in particularly handy when you want to eliminate objects from a raycast.  For example, if you want a cast to detect one type of object but not another while there might be multiple objects in the scene that a raycast would pick up normally. You may want to locate enemy players in a scene but totally ignore the players on your own team even if they are standing between your gun and the others. To achieve this, you put the members of your team on one layer and the members of the other team on another layer.  Then you add a mask to the raycast to avoid the members on your team’s layer.

Let’s try this out with our little scene.

To the raycasting cube add the following C# code:

using System.Collections.Generic;
using UnityEngine;

public class RaycastLayers : MonoBehaviour {
   
    void Update ()
    {
        RaycastHit hit;
        if (Physics.Raycast(transform.position, transform.TransformDirection(Vector3.forward), out hit, Mathf.Infinity))
        {
            Debug.DrawRay(transform.position, transform.TransformDirection(Vector3.forward) * hit.distance, Color.red);
            Debug.Log(“Hit: “ + hit.collider.gameObject.name);
        }
        else
        {
            Debug.DrawRay(transform.position, transform.TransformDirection(Vector3.forward) * 1000, Color.green);
            Debug.Log(“Missed.”);
        }
    }
}

Press play and a red line will be emitted from the raycast cube and hit the sphere. You’ll also get a message about hitting the sphere in the Console.

This code is doing nothing special but casting a ray out from the raycaster cube. When that raycast hits something it reports back on the object hit.

What happens if you move the sphere out of the road? The raycast continues until it hits something else.  Note that to hit something the ray must intersect with a collider.

Now, let’s assume you want the raycast to ignore the sphere completely.  Well you could put the sphere on the special layer called Ignore Raycast.  But then all rays in the environment would ignore the sphere and you might not want this.

So instead you can tell any raycast to ignore any layer you like. Assuming your sphere is on Layer 9 like mine, the code gets amended to (new code in bold):

…
void Update () 
{
    int layerMask = 1 << 9;
    layerMask = ~layerMask;

    RaycastHit hit;
    if (Physics.Raycast(transform.position, transform.TransformDirection(Vector3.forward), out hit, Mathf.Infinity, layerMask))
    {
        Debug.DrawRay(transform.position,
… 

Press play and with the sphere between the cube and the raycast cube, the red raycast line will go straight through the sphere and hit the cube, thus ignoring it.

So we’ve masked out the 9thlayer by adding in our own layer mask to the raycast.

What if you wanted to ignore the sphere and the cube?  Well assuming the sphere is on layer 9 and the cube is on layer 10.  The code becomes:

…
    void Update () 
    {
        int layerMaskSphere = 1 << 9;
        int layerMaskCube = 1 << 10;

        int layerMask = ~(layerMaskSphere | layerMaskCube);

        RaycastHit hit;
…

For each layer, you want to ignore you create another integer mask for it that is one bit shifted by the layer number then before a binary OR between them, e.g.

int layerMask = ~(layerMask1 | layerMask2 | layerMask3 | layerMask4);

Of course, you can always reverse this in the case you want to include rather than exclude a layer by removing the binary inverter “~” like this:

int layerMask = layerMask1 | layerMask2 | layerMask3 | layerMask4;

For further information see: