{"id":81607,"date":"2018-11-13T21:44:42","date_gmt":"2018-11-13T21:44:42","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=81607"},"modified":"2021-07-29T19:44:09","modified_gmt":"2021-07-29T19:44:09","slug":"voice-commands-in-unity","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/development\/dotnet-development\/voice-commands-in-unity\/","title":{"rendered":"Voice Commands in Unity"},"content":{"rendered":"

You don’t see it very often, but voice commands are no stranger to the world of video games. Games that could be played using your voice have existed since the late 90s with games like Hey You, Pikachu and Seaman being two notable examples from that time. Even now, with a little searching, you can easily find a game online that requires a microphone and your voice to play. What if you were told that you can make your own voice-controlled experience? As long as you have Unity and a microphone to test the project, you can!<\/p>\n

In a moment you’ll be creating a project that will be controlled using nothing but your voice. You will, of course, need a mic to be able to run the project. A single cube will be created, and you will be able to command the cube to change colors, spin in a certain direction, make a sound, and print a message to Unity’s Debug Log.<\/em> Accomplishing this will require Unity to look for certain phrases, which you will define. If what you say matches the phrase you define in code, then a user-defined function will be performed.<\/p>\n

Setting Up<\/h3>\n

Upon starting Unity, you will need to create a new project.<\/p>\n

<\/p>\n

Figure 1: Creating a new project.<\/p>\n

Give the project the name VoiceProject<\/em>, then specify the project location. This example images will be of a 3D project, but you can apply the same concepts in a 2D project as well. Once everything is set up, click Create Project<\/em>.<\/p>\n

<\/p>\n

Figure 2: Naming the project.<\/p>\n

The first thing you’ll be doing is creating the Cube <\/em>object needed for the project. In the Hierarchy <\/em>window click Create->3D Object->Cube. <\/em>For this project, you can leave the object to the default Cube <\/em>name.<\/p>\n

<\/p>\n

Figure 3: Creating a cube object.<\/p>\n

Once the object has been created, you’ll need to add an Audio Source <\/em>component to it. In the Inspector <\/em>window, click the Add Component <\/em>button. Search for Audio Source<\/em> then select the component at the top of the list.<\/p>\n

<\/p>\n

Figure 4: Adding a new component.<\/p>\n

Of course, an Audio Source <\/em>component would be rather useless without sounds. You can import sounds from your computer if you wish, but in this example, the Asset Store <\/em>will be used to acquire some free sounds. At the top of the Unity window select Window->General->Asset Store<\/em>.<\/p>\n

<\/p>\n

Figure 5: Opening the Asset Store.<\/em><\/p>\n

In the window that appears, search for Voices SFX <\/em>and select the corresponding item by Little Robot Sound Factory<\/em> in the drop-down menu that appears.<\/p>\n

<\/p>\n

Figure 6: Selecting Voices SFX.<\/em><\/p>\n

On the next screen, you’ll need to select the Download<\/em> button to download the assets. After the download has been completed, you’ll need to press the same button to import the assets. The button should say Import <\/em>after the download has completed.<\/p>\n

<\/p>\n

Figure 7: Beginning the import process.<\/p>\n

After pressing Import, <\/em>the Import Unity Package<\/em> window will appear. You can deselect all the sounds you don’t desire if you wish, but to keep things simple, this example import all the sounds in the package.<\/p>\n

<\/p>\n

Figure 8: Importing the assets.<\/p>\n

Once the sound assets have finished importing, you can either remove the Asset Store <\/em>window you opened or simply switch over to the Scene <\/em>window. Next, it will be time to make the script needed to implement voice commands. In the Assets <\/em>window, right-click and choose Create->C# Script<\/em>.<\/p>\n

<\/p>\n

Figure 9: Creating a new C# script<\/p>\n

Name this script VoiceControl.<\/em> When finished, the Assets <\/em>window will look like the below figure.<\/p>\n

<\/p>\n

Figure 10: The current Assets <\/em>window.<\/p>\n

Finally, attach the VoiceControl <\/em>script to the Cube <\/em>object. Select Cube <\/em>in the Hierarchy<\/em>, then click and drag the VoiceControl<\/em> script into the Inspector <\/em>window underneath the Add Component <\/em>button.<\/p>\n

<\/p>\n

Figure 11: Adding the VoiceControl <\/em>script component.<\/p>\n

Now that the script is attached to the object, it’s time to make the code. Open the script in Visual Studio by double-clicking it in the Assets <\/em>window.<\/p>\n

The Code<\/h2>\n

Before creating any voice commands, you will need the following using statements at the top of the script.<\/p>\n

using System.Collections.Generic;\r\nusing System.Linq;\r\nusing System;\r\nusing System.Collections;\r\nusing UnityEngine;\r\nusing UnityEngine.Windows.Speech;<\/pre>\n
The key statement is using UnityEngine.Windows.Speech<\/code>. As you may have guessed, this is what will allow Unity to take voice commands and perform certain actions from there. With that completed, declare the following variables inside the class.<\/p>\n
\/\/ Voice command vars\r\nprivate Dictionary<string, Action> keyActs = new Dictionary<string, Action>();\r\nprivate KeywordRecognizer recognizer;\r\n\/\/ Var needed for color manipulation\r\nprivate MeshRenderer cubeRend;\r\n\/\/Var needed for spin manipulation\r\nprivate bool spinningRight;\r\n\/\/Vars needed for sound playback.\r\nprivate AudioSource soundSource;\r\npublic AudioClip[] sounds;<\/pre>\n
First, you’ll need to define the Dictionary<\/code> that will store the voice commands and what action they perform. Next, you declare a KeywordRecognizer<\/code> to, well, recognize your words. Next, a MeshRenderer<\/code> variable is declared. This will get the MeshRenderer<\/code> component from the Cube<\/code> object. This is needed because it’s the MeshRenderer<\/code> that will allow you to change the color of the Cube<\/code> object. After that, you have a boolean<\/code> named spinningRight<\/code>. <\/em>You’ll use this boolean<\/code> to tell the program whether the Cube<\/code> object is to be spinning left or right depending on if spinningRight<\/code> is true or false. Next, you will create a private AudioSource<\/code> variable and a public array of AudioClip<\/code>. SoundSource<\/code> will be used to play sounds after hearing certain voice commands, and sounds<\/code> will simply be a list of sounds that can be played. You will define what goes into the sounds<\/code> array after entering the code. When you’ve entered all this, your script should look similar to the figure below.<\/p>\n
<\/p>\n
Figure 12: Using statements and variable declaration.<\/p>\n
From here, you’ll move on to the Start<\/code> function. The Update<\/code> function will not be needed for this project, so you can either comment it out or delete it. In the Start<\/code> function, input the following code:<\/p>\n
cubeRend = GetComponent<MeshRenderer>();\r\nsoundSource = GetComponent<AudioSource>();\r\n\/\/Voice commands for changing color\r\nkeyActs.Add(\"red\", Red);\r\nkeyActs.Add(\"green\", Green);\r\nkeyActs.Add(\"blue\", Blue);\r\nkeyActs.Add(\"white\", White);\r\n\/\/Voice commands for spinning\r\nkeyActs.Add(\"spin right\", SpinRight);\r\nkeyActs.Add(\"spin left\", SpinLeft);\r\n\/\/Voice commands for playing sound\r\nkeyActs.Add(\"please say something\", Talk);\r\n\/\/Voice command to show how complex it can get.\r\nkeyActs.Add(\"pizza is a wonderful food that makes the world better\", FactAcknowledgement);\r\nrecognizer = new KeywordRecognizer(keyActs.Keys.ToArray());\r\nrecognizer.OnPhraseRecognized += OnKeywordsRecognized;\r\nrecognizer.Start();<\/pre>\n
The Start<\/code> function can be split into three parts. First, you begin by getting the Cube<\/code> object’s MeshRenderer<\/code> and AudioSource<\/code> components using GetComponent<\/code>. After that, you will define the various voice commands and corresponding functions that will be in your keyActs<\/code> dictionary. You’ll be working on the functions in a moment. Notice how complex the voice commands can get. Near the end of your dictionary definitions, there’s an especially long voice command that is declared. As you’ll find out later, Unity will still be able to recognize this long series of words and perform the function that correlates to that command.<\/p>\n
Next, recognizer<\/code> gets its list of words to look out for by looking at the keys, or voice commands, in the keyActs<\/code> dictionary. Then, whenever a phrase is recognized, the OnKeywordsRecognized<\/code> function will be called, which will also be defined momentarily. Finally, the KeywordRecognizer<\/code> will be initialized and will continue to run for as long as the program is operational.<\/p>\n
When finished, the Start<\/code> function should appear similar to this:<\/p>\n
<\/p>\n
Figure 13: The completed Start <\/em>function.<\/p>\n
Now would be a good time to declare the different functions this script will need. OnKeywordsRecognized<\/code> will be a good place to begin. Add the following function:<\/p>\n
void OnKeywordsRecognized(PhraseRecognizedEventArgs args)\r\n{\r\n\tDebug.Log(\"Command: \" + args.text);\r\n\tkeyActs[args.text].Invoke();\r\n}<\/pre>\n
First, Unity’s console log will show what command was said once the user says any of the voice commands you’ve defined. Then the actual function that changes the object’s color will be called using Invoke<\/code>. After this, enter the code that will change the cube’s color.<\/p>\n
void Red()\r\n{\r\n\tcubeRend.material.SetColor(\"_Color\", Color.red);\r\n}\r\nvoid Green()\r\n{\r\n\tcubeRend.material.SetColor(\"_Color\", Color.green);\r\n}\r\nvoid Blue()\r\n{\r\n\tcubeRend.material.SetColor(\"_Color\", Color.blue);\r\n}\r\nvoid White()\r\n{\r\n\tcubeRend.material.SetColor(\"_Color\", Color.white);\r\n}<\/pre>\n
All of these functions perform the same basic task. They change the color of the cube object. The color the cube is changed to is dependent on the function. In each function, you first get the cubeRend's<\/code> material and call SetColor<\/code>. You then specify that you want to change the main color the material, and then specify the color in question. With that completed, your color changing functions and keyword recognizer function will look like Figure #.<\/p>\n
<\/p>\n
Figure 14: Color changing functions.<\/p>\n
Now you’ll work on the functions that spin the cube either left or right.<\/p>\n
void SpinRight()\r\n{\r\n\tspinningRight = true;\r\n\tStartCoroutine(RotateObject(1f));\r\n}\r\nvoid SpinLeft()\r\n{\r\n\tspinningRight = false;\r\n\tStartCoroutine(RotateObject(1f));\r\n}<\/pre>\n
These functions get a little more complex. First, you define whether spinningRight<\/code> is true or false, depending on which direction the Cube<\/code> object will be spinning. Then a Coroutine<\/code> is started that will rotate the object for one second. A Coroutine<\/code> is similar to a function, but it has the ability to pause execution and return control to Unity and then continue where it left off on the next frame. They’re very useful for actions such as spinning an object around. Coroutines<\/code> are declared with a return type of Ienumerator<\/code> and must contain a yield return statement somewhere in the body. The Coroutine<\/code> you’ll need is as follows:<\/p>\n
private IEnumerator RotateObject(float duration)\r\n{\r\n\tfloat startRot = transform.eulerAngles.x;\r\n\tfloat endRot;\r\n\tif (spinningRight)\r\n\t\tendRot = startRot - 360f;\r\n\telse\r\n\t\tendRot = startRot + 360f;\r\n\tfloat t = 0f;\r\n\tfloat yRot;\r\n\twhile (t < duration)\r\n\t{\r\n\t\tt += Time.deltaTime;\r\n\t\tyRot = Mathf.Lerp(startRot, endRot, t \/ duration) % 360.0f;\r\n\t\ttransform.eulerAngles = new Vector3(transform.eulerAngles.x, yRot, transform.eulerAngles.z);\r\n\t\tyield return null;\r\n\t}\r\n}<\/pre>\n
As mentioned earlier, it’s given a parameter that controls how long to spin the object for. In this case, it’s a float<\/code> variable named duration<\/code>. You then declare a few variables to be used within the Coroutine<\/code> along with their declarations. In the case of endRot<\/code>, it will change its starting value based on if spinningRight<\/code> is true or false. The while<\/code> loop will run for as long as t<\/code> is less than duration<\/code>. During this time, the object will be spun around by setting a new Vector3<\/code> to the object’s transform.eulerAngles<\/code>, <\/em>or rotation. <\/em>Mathf.Lerp<\/code> exists to help make the rotation look nice and smooth instead of jagged and ugly looking.<\/p>\n
The finished functions and Coroutine<\/code> should look like the figure below.<\/p>\n
<\/p>\n
Figure 15: Spinning functions and Coroutine<\/em>.<\/p>\n
Now you will make the program respond to your request to speak! Unfortunately, this program won’t allow you to have a complete conversation with the computer, but it could certainly be used as a starting point. Beneath the Coroutine<\/code>,<\/em> you will enter the following code.<\/p>\n
void Talk()\r\n{\r\n\tsoundSource.clip = sounds[UnityEngine.Random.Range(0, sounds.Length)];\r\n\tsoundSource.Play();\r\n}<\/pre>\n
After responding to the command \u201cplease say something,\u201d the program will respond by playing a random sound. You get the sound clip from the sounds<\/code> array and then immediately play the sound afterward. Finally, create one last function to work with the last voice command you created.<\/p>\n
void FactAcknowledgement()\r\n{\r\n\tDebug.Log(\"How right you are.\");\r\n}<\/pre>\n
There’s not much going on in this function. After registering the voice command, Unity simply prints a message to the Debug Log<\/em> agreeing with your statement. With those two functions out of the way, the script is now complete. Save your work and return to the Unity editor to finish the project.<\/p>\n
<\/p>\n
Figure 16: The Talk <\/em>and FactAcknowledgement <\/em>functions.<\/strong><\/p>\n
Completing the Project<\/h2>\n
One task must be completed before you can test out your project. You need to assign some sound effects to the sounds <\/em>array. To do this, select the Cube, go to the Inspector <\/em>window and into the VoiceControl <\/em>script component. Then, click the arrow next to the sounds <\/em>array.<\/p>\n
<\/p>\n
Figure 17: Opening the sounds <\/em>array.<\/p>\n
A field named Size <\/em>appears when you click this arrow. Decide on how many individual sound effects you’d like to use. The example figure below sets the value of Size <\/em>to five.<\/p>\n
<\/p>\n
Figure 18: Defining the number of elements.<\/p>\n
The moment you give Size <\/em>a number, a list of empty elements appears. At this point, you need to drag some sound assets into these fields to populate the array. Remember those sound effects you downloaded early on? They’re going to be put to use here. In the Assets <\/em>window, navigate to Voices SFX->Mp3<\/em>. From there you must select one of the folders that contain the sounds you want. Any of the sound effects will do, but this example will use the sounds in the Robot <\/em>folder.<\/p>\n
At this point, you’ll need to select the Cube <\/em>object from the Hierarchy <\/em>and lock the inspector so you can more easily set the sounds to be used in your sounds <\/em>array. With the Cube <\/em>object selected, click the lock icon in the top right of the Inspector <\/em>window.<\/p>\n
<\/p>\n
Figure 19: Locking the Inspector<\/em>.<\/p>\n
All you need to do now is click and drag whatever sounds you wish to use into the empty fields in the sounds <\/em>array.<\/p>\n
<\/p>\n
Figure 20: Setting sound effects for sounds <\/em>array.<\/p>\n
After selecting your sounds of choice, the project will be complete! Give the project a test run by pressing the play button at the top of the Unity window. While playing, say any of the voice commands and watch your object change and react based on those commands. Don’t forget to have your microphone ready! Below is a list of the voice commands you’ve made:<\/p>\n
    \n
  • \u201cRed\u201d<\/li>\n
  • \u201cBlue\u201d<\/li>\n
  • \u201cGreen\u201d<\/li>\n
  • \u201cSpin Left\u201d<\/li>\n
  • \u201cSpin Right\u201d<\/li>\n
  • \u201cPlease say something\u201d<\/li>\n
  • \u201cPizza is a wonderful food that makes the world better\u201d<\/li>\n<\/ul>\n
    <\/p>\n
    Figure 21: The project in action.<\/p>\n
    Conclusion<\/h2>\n
    Voice commands don’t have to be relegated to mere gimmicks. The examples mentioned at the beginning are two examples of games using the voice to play. Though they aren’t very common, there are plenty of other cases of voice commands in video games. You can certainly take that functionality outside of gaming as well. Most smartphones have voice input functionality allowing you to create text messages or check the weather. Here, you made an object change its color, spin around, play a sound, and print a message to the console using Debug Log. <\/em>This is all being done with your voice. No hands required!<\/p>\n
    At the end of the day, the microphone can be used as another input device just like a mouse or keyboard if you know how to utilize it. Perhaps it can be used for accessibility, or to assist in multi-tasking. Voice commands are often relegated to mere gimmicks, but with the right idea and ability to do it, the power of the voice can be much better realized in games and other applications. Perhaps you have that idea. You can make it a reality!<\/p>\n","protected":false},"excerpt":{"rendered":"
    Today, we use voice in many ways. We can order groceries, ask to hear certain musical recordings, and much more with voice commands to our devices. In this article, Lance Talbert demonstrates how to add voice commands to video games.…<\/p>\n","protected":false},"author":317499,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[143538,53],"tags":[],"coauthors":[52549],"class_list":["post-81607","post","type-post","status-publish","format-standard","hentry","category-dotnet-development","category-featured"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/81607","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/317499"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=81607"}],"version-history":[{"count":7,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/81607\/revisions"}],"predecessor-version":[{"id":81720,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/81607\/revisions\/81720"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=81607"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=81607"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=81607"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=81607"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}