Magic Leap Patent | Method of waking a device using spoken voice commands
Publication Number: 20250168567
Publication Date: 2025-05-22
Assignee: Magic Leap
Abstract
Disclosed herein are systems and methods for processing speech signals in mixed reality applications. A method may include receiving an audio signal; determining, via first processors, whether the audio signal comprises a voice onset event; in accordance with a determination that the audio signal comprises the voice onset event: waking a second one or more processors; determining, via the second processors, that the audio signal comprises a predetermined trigger signal; in accordance with a determination that the audio signal comprises the predetermined trigger signal: waking third processors; performing, via the third processors, automatic speech recognition based on the audio signal; and in accordance with a determination that the audio signal does not comprise the predetermined trigger signal: forgoing waking the third processors; and in accordance with a determination that the audio signal does not comprise the voice onset event: forgoing waking the second processors.
Claims
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a Continuation of U.S. Non-Provisional application Ser. No. 18/418,131, filed Jan. 19, 2024, which is a Continuation of U.S. Non-Provisional application Ser. No. 17/214,446, filed Mar. 26, 2021, which claims benefit of U.S. Provisional Application No. 63/001,116, filed Mar. 27, 2020, and U.S. Provisional Application No. 63/033,451, filed Jun. 2, 2020, the contents of which are incorporated herein by reference in their entirety.
FIELD
This disclosure relates in general to systems and methods for processing speech signals, and in particular to systems and methods for processing speech signals in a mixed reality environment.
BACKGROUND
Systems for speech recognition are tasked with receiving audio input representing human speech, typically via one or more microphones, and processing the audio input to determine words, logical structures, or other outputs corresponding to that audio input. For example, automatic speech recognition (ASR) systems may generate a text output based on the human speech corresponding to an audio input signal; and natural language processing (NLP) tools may generate logical structures, or computer data, corresponding to the meaning of that human speech. While such systems may contain any number of components, at the heart of such systems is a speech processing engine, which is a component that accepts an audio signal as input, performs some recognition logic on the input, and outputs some text corresponding to that input. (While reference is made herein to speech processing engines, other forms of speech processing besides speech recognition should also be considered within the scope of the disclosure.)
Historically, audio input was provided to speech processing engines in a structured, predictable manner. For example, a user might speak directly into a microphone of a desktop computer in response to a first prompt (e.g., “Begin Speaking Now”); immediately after pressing a first button input (e.g., a “start” or “record” button, or a microphone icon in a software interface); or after a significant period of silence. Similarly, a user might stop providing microphone input in response to a second prompt (e.g., “Stop Speaking”); immediately before pressing a second button input (e.g., a “stop” or “pause” button); or by remaining silent for a period of time. Such structured input sequences left little doubt as to when the user was providing input to a speech processing engine (e.g., between a first prompt and a second prompt, or between pressing a start button and pressing a stop button). Moreover, because such systems typically required deliberate action on the part of the user, it could generally be assumed that a user's speech input was directed to the speech processing engine, and not to some other listener (e.g., a person in an adjacent room). Accordingly, many speech processing engines of the time may not have had any particular need to identify, from microphone input, which portions of the input were directed to the speech processing engine and were intended to provide speech recognition input, and conversely, which portions were not.
The ways in which users provide speech recognition input has changed as speech processing engines have become more pervasive and more fully integrated into users' everyday lives. For example, some automated voice assistants are now housed in or otherwise integrated with household appliances, automotive dashboards, smart phones, wearable devices, “living room” devices (e.g., devices with integrated “smart” voice assistants), and other environments far removed from the conventional desktop computer. In many cases, speech processing engines are made more broadly usable by this level of integration into everyday life. However, these systems would be made cumbersome by system prompts, button inputs, and other conventional mechanisms for demarcating microphone input to the speech processing engine. Instead, some such systems place one or more microphones in an “always on” state, in which the microphones listen for a “wake-up word” (e.g., the “name” of the device or any other predetermined word or phrase) that denotes the beginning of a speech recognition input sequence. Upon detecting the wake-up word, the speech processing engine can process the following sequence of microphone input as input to the speech processing engine.
While the wake-up word system replaces the need for discrete prompts or button inputs for speech processing engines, it can be desirable to minimize the amount of time the wake-up word system is required to be active. For example, mobile devices operating on battery power benefit from both power efficiency and the ability to invoke a speech processing engine (e.g., invoking a smart voice assistant via a wake-up word). For mobile devices, constantly running the wake-up word system to detect the wake-up word may undesirably reduce the device's power efficiency. Ambient noises or speech other than the wake-up word may be continually processed and transcribed, thereby continually consuming power. However, processing and transcribing ambient noises or speech other than the wake-up word may not justify the required power consumption. It therefore can be desirable to minimize the amount of time the wake-up word system is required to be active without compromising the device's ability to invoke a speech processing engine.
In addition to reducing power consumption, it is also desirable to improve the accuracy of speech recognition systems. For example, a user who wishes to invoke a smart voice assistant may become frustrated if the smart voice assistant does not accurately respond to the wake-up word. The smart voice assistant may respond to an acoustic event that is not the wake-up word (i.e., false positives), the assistant may fail to respond to the wake-up word (i.e., false negatives), or the assistant may respond too slowly to the wake-up word (i.e., lag). Inaccurate responses to the wake-up word like the above examples may frustrate the user, leading to a degraded user experience. The user may further lose trust in the reliability of the product's speech processing engine interface. It therefore can be desirable to develop a speech recognition system that accurately responds to user input.
BRIEF SUMMARY
Examples of the disclosure describe systems and methods for processing speech signals in mixed reality applications. According to examples of the disclosure, a method may include receiving, via a microphone, an audio signal; determining, via a first one or more processors, whether the audio signal comprises a voice onset event; in accordance with a determination that the audio signal comprises the voice onset event: waking a second one or more processors; determining, via the second one or more processors, that the audio signal comprises a predetermined trigger signal; in accordance with a determination that the audio signal comprises the predetermined trigger signal: waking a third one or more processors; performing, via the third one or more processors, automatic speech recognition based on the audio signal; and in accordance with a determination that the audio signal does not comprise the predetermined trigger signal: forgoing waking the third one or more processors; and in accordance with a determination that the audio signal does not comprise the voice onset event: forgoing waking the second one or more processors.
In some embodiments, a method comprises: receiving, via a first microphone, an audio signal; determining, via a first one or more processors, whether the audio signal comprises a voice onset event; in accordance with a determination that the audio signal comprises the voice onset event: waking a second one or more processors; determining, via the second one or more processors, whether the audio signal comprises a predetermined trigger signal; in accordance with a determination that the audio signal comprises the predetermined trigger signal: waking a third one or more processors; performing, via the third one or more processors, automatic speech recognition based on the audio signal; and in accordance with a determination that the audio signal does not comprise the predetermined trigger signal: forgoing waking the third one or more processors; and in accordance with a determination that the audio signal does not comprise the voice onset event: forgoing waking the second one or more processors.
In some embodiments, the method further comprises: in accordance with the determination that the audio signal comprises the predetermined trigger signal: providing, via the second one or more processors, an audio stream to the third one or more processors based on the audio signal; identifying an endpoint corresponding to the audio signal; and ceasing to provide the audio stream to the third one or more processors in response to identifying the endpoint.
In some embodiments, the first one or more processors comprises an application-specific integrated circuit or a digital signal processor configured to determine whether the audio signal comprises the voice onset event.
In some embodiments, the second one or more processors comprises a digital signal processor or an application-specific integrated circuit.
In some embodiments, the third one or more processors comprises a general purpose processor.
In some embodiments, the first one or more processors and the second one or more processors are implemented on an application-specific integrated circuit.
In some embodiments, a head-wearable device comprises the first one or more processors and the second one or more processors, wherein an auxiliary unit comprises the third one or more processors, and wherein the auxiliary unit is external to the head-wearable device and configured to communicate with the head-wearable device.
In some embodiments, the predetermined trigger signal comprises a phrase.
In some embodiments, the method further comprises storing the audio signal in a buffer.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, acoustic echo cancellation based on the audio signal.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, beamforming based on the audio signal.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, noise reduction based on the audio signal.
In some embodiments, the method further comprises receiving, via a second microphone, the audio signal, wherein the determination whether the audio signal comprises the voice onset event is based on an output of the first microphone and further based on an output of the second microphone.
In some embodiments, the audio signal includes a signal generated by a voice source of a user of a device comprising the first microphone and further comprising the second microphone, and a first distance from the first microphone to the voice source and a second distance from the second microphone to the voice source are different.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining an amount of voice activity in the audio signal.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining whether the amount of voice activity in the audio signal is greater than a voice activity threshold.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining a first probability of voice activity with respect to the audio signal.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises: determining a second probability of voice activity with respect to the audio signal; and determining a combined probability of voice activity based on the first probability of voice activity and further based on the second probability of voice activity.
In some embodiments, the first probability is associated with a single channel signal of the audio signal, and the second probability is associated with a beamforming signal of the audio signal.
In some embodiments, determining the first probability of voice activity with respect to the audio signal comprises computing a ratio associated with the audio signal and calculating the first probability by inputting the ratio to a voice activity probability function.
In some embodiments, the voice activity probability function is tuned using at least one of user input, voice activity statistics, neural network input, and machine learning.
In some embodiments, the audio signal includes a single channel signal, and the ratio is an input power to noise power ratio.
In some embodiments, the audio signal includes a beamforming signal, and the ratio is a ratio of a summation signal of the audio signal and a normalized different signal of the audio signal.
In some embodiments, in accordance with the determination that the audio signal comprises the voice onset event, the method further comprises evaluating an amount of voice activity prior to a time associated with the determination that the audio signal comprises the voice onset event.
In some embodiments, in accordance with the determination that the audio signal comprises the voice onset event, the method further comprises pausing determining whether a second audio signal comprises a voice onset event for an amount of time.
In some embodiments, a system comprises: a first microphone; one or more processors configured to execute a method comprising: receiving, via the first microphone, an audio signal; determining, via a first one or more processors, whether the audio signal comprises a voice onset event; in accordance with a determination that the audio signal comprises the voice onset event: waking a second one or more processors; determining, via the second one or more processors, whether the audio signal comprises a predetermined trigger signal; in accordance with a determination that the audio signal comprises the predetermined trigger signal: waking a third one or more processors; performing, via the third one or more processors, automatic speech recognition based on the audio signal; and in accordance with a determination that the audio signal does not comprise the predetermined trigger signal: forgoing waking the third one or more processors; and in accordance with a determination that the audio signal does not comprise the voice onset event: forgoing waking the second one or more processors.
In some embodiments, the method further comprises: in accordance with the determination that the audio signal comprises the predetermined trigger signal: providing, via the second one or more processors, an audio stream to the third one or more processors based on the audio signal; identifying an endpoint corresponding to the audio signal; and ceasing to provide the audio stream to the third one or more processors in response to identifying the endpoint.
In some embodiments, the first one or more processors comprises an application-specific integrated circuit or a digital signal processor configured to determine whether the audio signal comprises the voice onset event.
In some embodiments, the second one or more processors comprises a digital signal processor or an application-specific integrated circuit.
In some embodiments, the third one or more processors comprises a general purpose processor.
In some embodiments, the first one or more processors and the second one or more processors are implemented on an application-specific integrated circuit.
In some embodiments, the system further comprises: a head-wearable device comprising the first one or more processors and the second one or more processors, and an auxiliary unit comprising the third one or more processors, wherein the auxiliary unit is external to the head-wearable device and configured to communicate with the head-wearable device.
In some embodiments, the predetermined trigger signal comprises a phrase.
In some embodiments, the method further comprises storing the audio signal in a buffer.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, acoustic echo cancellation based on the audio signal.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, beamforming based on the audio signal.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, noise reduction based on the audio signal.
In some embodiments, the method further comprises receiving, via a second microphone, the audio signal, wherein the determination whether the audio signal comprises the voice onset event is based on an output of the first microphone and further based on an output of the second microphone.
In some embodiments, the audio signal includes a signal generated by a voice source of a user of a device comprising the first microphone and further comprising the second microphone, and a first distance from the first microphone to the voice source and a second distance from the second microphone to the voice source are different.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining an amount of voice activity in the audio signal.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining whether the amount of voice activity in the audio signal is greater than a voice activity threshold.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining a first probability of voice activity with respect to the audio signal.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises: determining a second probability of voice activity with respect to the audio signal; and determining a combined probability of voice activity based on the first probability of voice activity and further based on the second probability of voice activity.
In some embodiments, the first probability is associated with a single channel signal of the audio signal, and the second probability is associated with a beamforming signal of the audio signal.
In some embodiments, determining the first probability of voice activity with respect to the audio signal comprises computing a ratio associated with the audio signal and calculating the first probability by inputting the ratio to a voice activity probability function.
In some embodiments, the voice activity probability function is tuned using at least one of user input, voice activity statistics, neural network input, and machine learning.
In some embodiments, the audio signal includes a single channel signal, and the ratio is an input power to noise power ratio.
In some embodiments, the audio signal includes a beamforming signal, and the ratio is a ratio of a summation signal of the audio signal and a normalized different signal of the audio signal.
In some embodiments, the method further comprises in accordance with the determination that the audio signal comprises the voice onset event, further comprising evaluating an amount of voice activity prior to a time associated with the determination that the audio signal comprises the voice onset event.
In some embodiments, the method further comprises in accordance with the determination that the audio signal comprises the voice onset event, further comprising pausing determining whether a second audio signal comprises a voice onset event for an amount of time.
In some embodiments, a non-transitory computer-readable medium stores one or more instructions, which, when executed by one or more processors of an electronic device, cause the device to perform a method comprising: receiving, via a first microphone, an audio signal; determining, via a first one or more processors, whether the audio signal comprises a voice onset event; in accordance with a determination that the audio signal comprises the voice onset event: waking a second one or more processors; determining, via the second one or more processors, whether the audio signal comprises a predetermined trigger signal; in accordance with a determination that the audio signal comprises the predetermined trigger signal: waking a third one or more processors; performing, via the third one or more processors, automatic speech recognition based on the audio signal; and in accordance with a determination that the audio signal does not comprise the predetermined trigger signal: forgoing waking the third one or more processors; and in accordance with a determination that the audio signal does not comprise the voice onset event: forgoing waking the second one or more processors.
In some embodiments, the method further comprises: in accordance with the determination that the audio signal comprises the predetermined trigger signal: providing, via the second one or more processors, an audio stream to the third one or more processors based on the audio signal; identifying an endpoint corresponding to the audio signal; and ceasing to provide the audio stream to the third one or more processors in response to identifying the endpoint.
In some embodiments, the first one or more processors comprises an application-specific integrated circuit or a digital signal processor configured to determine whether the audio signal comprises the voice onset event.
In some embodiments, the second one or more processors comprises a digital signal processor or an application-specific integrated circuit.
In some embodiments, the third one or more processors comprises a general purpose processor.
In some embodiments, the first one or more processors and the second one or more processors are implemented on an application-specific integrated circuit.
In some embodiments, a head-wearable device comprises the first one or more processors and the second one or more processors, wherein an auxiliary unit comprises the third one or more processors, and wherein the auxiliary unit is external to the head-wearable device and configured to communicate with the head-wearable device.
In some embodiments, the predetermined trigger signal comprises a phrase.
In some embodiments, the method further comprises storing the audio signal in a buffer.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, acoustic echo cancellation based on the audio signal.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, beamforming based on the audio signal.
In some embodiments, the method further comprises: in accordance with a determination that the audio signal comprises the voice onset event: performing, via the second one or more processors, noise reduction based on the audio signal.
In some embodiments, the method further comprises receiving, via a second microphone, the audio signal, wherein the determination whether the audio signal comprises the voice onset event is based on an output of the first microphone and further based on an output of the second microphone.
In some embodiments, the audio signal includes a signal generated by a voice source of a user of a device comprising the first microphone and further comprising the second microphone, and a first distance from the first microphone to the voice source and a second distance from the second microphone to the voice source are different.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining an amount of voice activity in the audio signal.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining whether the amount of voice activity in the audio signal is greater than a voice activity threshold.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises determining a first probability of voice activity with respect to the audio signal.
In some embodiments, the determining whether the audio signal comprises the voice onset event further comprises: determining a second probability of voice activity with respect to the audio signal; and determining a combined probability of voice activity based on the first probability of voice activity and further based on the second probability of voice activity.
In some embodiments, the first probability is associated with a single channel signal of the audio signal, and the second probability is associated with a beamforming signal of the audio signal.
In some embodiments, determining the first probability of voice activity with respect to the audio signal comprises computing a ratio associated with the audio signal and calculating the first probability by inputting the ratio to a voice activity probability function.
In some embodiments, the voice activity probability function is tuned using at least one of user input, voice activity statistics, neural network input, and machine learning.
In some embodiments, the audio signal includes a single channel signal, and the ratio is an input power to noise power ratio.
In some embodiments, the audio signal includes a beamforming signal, and the ratio is a ratio of a summation signal of the audio signal and a normalized different signal of the audio signal.
In some embodiments, the method further comprises in accordance with the determination whether the audio signal comprises the voice onset event, further comprising evaluating an amount of voice activity prior to a time associated with the determination that the audio signal comprises the voice onset event.
In some embodiments, the method further comprises in accordance with the determination that the audio signal comprises the voice onset event, further comprising pausing determining whether a second audio signal comprises a voice onset event for an amount of time.
BRIEF DESCRIPTION OF THE DRAWINGS
FIGS. 1A-1B illustrate example wearable systems according to some embodiments of the disclosure.
FIG. 2 illustrates an example handheld controller that can be used in conjunction with an example wearable system according to some embodiments of the disclosure.
FIG. 3 illustrates an example auxiliary unit that can be used in conjunction with an example wearable system according to some embodiments of the disclosure.
FIGS. 4A-4B illustrate example functional block diagrams for an example wearable system according to some embodiments of the disclosure.
FIG. 5 illustrates a flow chart of an example system for determining an onset of voice activity according to some embodiments of the disclosure.
FIGS. 6A-6C illustrate examples of processing input audio signals according to some embodiments of the disclosure.
FIGS. 7A-7E illustrate examples of processing input audio signals according to some embodiments of the disclosure.
FIG. 8 illustrates an example of determining an onset of voice activity according to some embodiments of the disclosure.
FIG. 9 illustrates an example of determining an onset of voice activity according to some embodiments of the disclosure.
FIG. 10 illustrates an example MR system, according to some embodiments of the disclosure.
FIGS. 11A-11C illustrate example signal processing steps, according to some embodiments of the disclosure.
FIG. 12 illustrates an example MR computing architecture, according to some embodiments of the disclosure.
