Sony Patent | Information processing device and information processing method

In the case of the above equation (14), a vector distance (scalar value) between the normal respiration origin “O_N” and the respiratory state vector “H_v” on a (d_b, f_b) plane, which is the long span time observation result, is used as the value of the voice speech influence level “E_u”. Note that the equation (14) is merely an example, and the information processing system 1 may calculate the voice speech influence level “E_u” using various equations.

Then, the information processing system 1 determines the extended time length using the calculated voice speech influence level “E_u”. Note that, since this point is similar to the first embodiment, description is omitted. The information processing system 1 extends the silent timeout time “t_r” of the voice recognition speech end determination or the silent timeout time “t_s” of the voice interaction session end according to the respiratory state. The information processing system 1 extends the silent timeout times “t_r” and “t_s” longer as the value of the voice speech influence level “E_u” becomes larger. For example, the information processing system 1 extends the silent timeout times “t_r” and “t_s” by the time proportional to the voice speech influence level “E_u”. Details of the control flow of the silent timeout times “t_r” and “t_s” in normal times and during exercise will be described with reference to FIGS. 22A and 22B.

Furthermore, in the case of detection of the respiratory state vector “H_v”, the information processing system 1 extends the silent timeout times “t_r” and “t_s” until the speed “v_b” becomes smaller than a threshold “v_f” when the instantaneous respiration speed “v_b” is equal to or larger than (faster than) the threshold “v_f” under a condition of d_b≤d₀ or f_b≤f₀. Furthermore, the information processing system 1 extends the silent timeout times “t_r” and “t_s” until the speed “v_b” becomes larger than a threshold “v_s” when the speed “v_b” is equal to or smaller than (slower than) the threshold “v_s” under a condition of d_b≥d₀ or f_b≥f₀. The information processing system 1 extends the timeout times by a period in which the user cannot temporarily speak in a case where the speed of the respiration instantaneously increases due to a physiological phenomenon of a respiratory system or in a case where the speed of the respiration instantaneously decreases (stops) due to surprise or strain.

2-2. Configuration of Information Processing Device According to Second Embodiment

Next, a configuration of the server device 100A, which is an example of an information processing device that executes information processing according to the second embodiment, will be described. FIG. 10 is a diagram illustrating a configuration example of a server device according to the second embodiment of the present disclosure.

As illustrated in FIG. 10, the server device 100A includes a communication unit 110, a storage unit 120A, and a control unit 130A.

The storage unit 120A is implemented by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As illustrated in FIG. 10, the storage unit 120A according to the second embodiment includes a respiration information storage unit 121, a user information storage unit 122, a determination information storage unit 123A, and a functional information storage unit 124.

The determination information storage unit 123A according to the second embodiment stores various types of information regarding information (determining information) to be used for determination. The determination information storage unit 123A stores various types of information to be used for determining whether or not to execute the voice interaction control. FIG. 11 is a diagram illustrating an example of the determination information storage unit according to the second embodiment of the present disclosure. The determination information storage unit 123A illustrated in FIG. 11 includes items such as “determination information ID”, “use”, “name”, and “content”.

The “determination information ID” indicates identification information for identifying determination information. The “use” indicates use of the determination information. The “name” indicates a name (character string) of the determination information (variable) identified by the corresponding determination information ID. The “content” indicates specific content (value or the like) of the determination information identified by the corresponding determination information ID.

In the example of FIG. 11, the use of the determination information (determination information JD1) identified by the determination information ID “JD1” indicates the determination information to be used for determination of the normal range. The determination information JD1 indicates determination information (normal range) to be used for comparison with the respiratory state vector. The determination information JD1 indicates that the information is used as the name “R_N”. The content of the determination information JD1 indicates “range information AINF1”. Note that FIG. 11 illustrates the content with an abstract code such as “range information AINF1”, but the content is assumed to be specific information (vector, numerical value, or the like) such as “(1.2, 32, 2.8, . . . )” or “2.6”. For example, the “range information AINF1” may be information (numerical value) indicating a distance from the origin (for example, O_N) or may be N-dimensional vector information indicating a range.

Note that the determination information storage unit 123A may store various types of information according to a purpose, in addition to the above-described information.

As illustrated in FIG. 10, the control unit 130A includes an acquisition unit 131, a calculation unit 132A, a determination unit 133A, an execution unit 134, and a transmission unit 135, and implements or executes a function and an action of the information processing to be described below. The acquisition unit 131 acquires information from the determination information storage unit 123A.

The calculation unit 132A calculates various types of information similarly to the calculation unit 132. The calculation unit 132A calculates various types of information on the basis of the information in the determination information storage unit 123A. The calculation unit 132A calculates the vector indicating the respiratory state of the user using the respiration information. The calculation unit 132A calculates the respiratory state vector “H_v” that is a vector, using the equation (8) and the like. The calculation unit 132A calculates the depth “d_b”, using the equation (9). The calculation unit 132A calculates the average observation value “S_m”, using the equation (10). The calculation unit 132A detects a peak from a maximum value, a minimum value, or the like between intersections and “S_m”, and calculates a peak observation value “S_pi”. The calculation unit 132A calculates (counts) the number of peak observations “N_1p”. The calculation unit 132A calculates the speed “v_b”, using the equation (11). The calculation unit 132A calculates a difference value “ΔS_i”, using the equation (12).

The determination unit 133A determines various types of information similarly to the determination unit 133. The determination unit 133A determines various types of information on the basis of the information in the determination information storage unit 123A. The determination unit 133A determines whether or not to execute the voice interaction control by comparing the information calculated by the calculation unit 132 with the normal range. The determination unit 133A determines whether or not to execute the voice interaction control using the information of the normal range. The determination unit 133A determines whether or not to execute the voice interaction control by comparing respiratory state vector “H_v” with the normal range. The determination unit 133A determines to execute the voice interaction control in the case where the respiratory state vector “H_v” falls outside the normal range “R_N”.

The determination unit 133A determines whether or not to execute the voice interaction control by comparing the information calculated by the calculation unit 132 with the normal range. The determination unit 133A determines whether or not to execute the voice interaction control using the normal range. The determination unit 133A determines whether or not to execute the voice interaction control by comparing respiratory state vector “H_v” with the normal range. The determination unit 133A determines to execute the voice interaction control in the case where the respiratory state vector “H_v” falls outside the normal range “R_N”. The determination unit 133A compares the respiratory state vector “H_v” with the normal range “R_N”. The determination unit 133A determines that the respiratory state of the user is normal in the case where the respiratory state vector “H_v” is within normal range “R_N”. The determination unit 133A determines that the respiratory state of the user is non-normal in the case where the respiratory state vector “H_v” is out of the normal range “R_N”.

The execution unit 134 executes various types of processing similarly to the execution unit 134 according to the first embodiment. The execution unit 134 executes various types of processing on the basis of the information of the determination information storage unit 123A. The execution unit 134 executes the voice interaction control in a case where the determination unit 133A determines to execute the voice interaction control. In the case where the vector satisfies the condition, the execution unit 134 executes the processing of concatenating the first speech and the second speech by executing the voice interaction control. In the case where the vector falls outside the normal range, the execution unit 134 executes the processing of concatenating the first speech and the second speech by executing the voice interaction control.

The execution unit 134 executes voice interaction control in a case where the respiratory state vector “H_v” satisfies a condition. The execution unit 134 executes voice interaction control in a case where a comparison result between the respiratory state vector “H_v” and the normal range “R_N” satisfies a condition. The execution unit 134 executes the voice interaction control in the case where the respiratory state vector “H_v” falls outside the normal range “R_N”.

2-3. Procedure of Information Processing According to Second Embodiment

Next, an example of specific processing related to the information processing system will be described with reference to FIGS. 12A and 12B and FIG. 13. FIGS. 12A and 12B and FIG. 13 are flowcharts illustrating processing of the information processing system according to the second embodiment of the present disclosure. Specifically, FIGS. 12A, 12B, and 13 are flowcharts related to a voice interaction session. FIGS. 12A, 12B, and 13 illustrate a voice interaction control flowchart with the respiratory state vector “H_v” including an extended silent timeout time with the speed “v_b”. Note that, hereinafter, the case where the information processing system 1 according to the second embodiment performs the processing will be described as an example, but the processing illustrated in FIGS. 12A, 12B, and 13 may be performed by either the server device 100A or the terminal device 10 included in the information processing system 1 according to the second embodiment.

As illustrated in FIG. 12A, the information processing system 1 determines whether or not respiratory state vector “H_v” of the respiration falls outside the normal range “R_N” (step S401).

In the case where the respiratory state vector “H_v” of the respiration falls outside the normal range “R_N” (step S401: Yes), the information processing system 1 calculates the voice speech influence level “E_u” from the respiratory state vector “H_v” and the normal respiration origin “O_N” (step S402).

Then, the information processing system 1 extends the voice recognition timeout time “t_r” by the time proportional to the voice speech influence level “E_u” (step S403). Furthermore, the information processing system 1 extends the session timeout time “t_s” by the time proportional to the voice speech influence level “E_u” (step S404). Then, the information processing system 1 performs the processing of step S405 illustrated in FIG. 13. As described above, the example of FIGS. 12A, 12B, and 13 illustrates the case where the information processing system 1 extends the timeout time as the voice interaction control as an example.

Furthermore, in the case where the respiratory state vector “H_v” of the respiration does not fall outside the normal range “R_N” (step S401: No), the information processing system 1 executes the processing of step S405 illustrated in FIG. 13 without performing the processing of steps S402 to S404. For example, in the case where the respiratory state vector “H_v” of the respiration falls within the normal range “R_N”, the information processing system 1 executes the processing of step S405 illustrated in FIG. 13 without performing the processing of steps S402 to S404.

As illustrated in FIG. 13, the information processing system 1 determines whether or not the condition that the depth “d_b” of the respiratory state vector “H_v” is equal to or less than the depth “d₀” at the time of normal respiration or the frequency “f_b” is equal to or less than the frequency “f₀” at the time of normal respiration is satisfied (step S405). The information processing system 1 executes the processing of step S407 without performing the processing of step S406 in the case where the condition that the depth “d_b” of the respiratory state vector “H_v” is equal to or less than the depth “d₀” at the time of normal respiration or the frequency “f_b” is equal to or less than the frequency “f₀” at the time of normal respiration is not satisfied (step S405: No). For example, the information processing system 1 executes the processing of step S407 without performing the processing of step S406 in the case where the depth “d_b” of the respiratory state vector “H_v” is not equal to or less than the depth “d₀” of the normal respiration origin “O_N” or the frequency “f_b” is not equal to or less than the frequency “f₀” of the normal respiration origin “O_N”.

Meanwhile, the information processing system 1 determines whether or not the speed “v_b” of the respiratory state vector “H_v” is smaller than the threshold “v_f” (step S406) in the case where the condition that the depth “d_b” of the respiratory state vector “H_v” is equal to or less than the depth “d₀” at the time of normal respiration or the frequency “f_b” is equal to or less than the frequency “f₀” at the time of normal respiration (step S405: Yes).

The information processing system 1 determines whether or not the condition that the depth “d_b” of the respiratory state vector “H_v” is equal or larger than the depth “d₀” at the time of normal respiration or the frequency “f_b” is equal to or larger than the frequency “f₀” at the time of normal respiration is satisfied (step S407) in the case where the speed “v_b” of the respiratory state vector “H_v” is smaller than the threshold “v_f” (step S406: Yes).

The information processing system 1 executes the processing of step S409 without performing the processing of step S408 in the case where the condition that the depth “d_b” of the respiratory state vector “H_v” is equal or larger than the depth “d₀” at the time of normal respiration or the frequency “f_b” is equal to or larger than the frequency “f₀” at the time of normal respiration is not satisfied (step S407: No). For example, the information processing system 1 executes the processing of step S409 without performing the processing of step S408 in the case where the depth “d_b” of the respiratory state vector “H_v” is not equal or larger than the depth “d₀” of the normal respiration origin “O_N” or the frequency “f_b” is not equal to or larger than the frequency “f₀” of the normal respiration origin “O_N”.

Meanwhile, the information processing system 1 determines whether or not the speed “v_b” of the respiratory state vector “H_v” is larger than the threshold “v_s” (step S408) in the case where the condition that the depth “d_b” of the respiratory state vector “H_v” is equal or larger than the depth “d₀” at the time of normal respiration or the frequency “f_b” is equal to or larger than the frequency “f₀” at the time of normal respiration (step S407: Yes).

The information processing system 1 performs the processing of step S409 in the case where the speed “v_b” of the respiratory state vector “H_v” is larger than the threshold “v_s” (step S408: Yes).

The information processing system 1 determines whether or not the session timeout time “t_s” has elapsed without a speech or the voice recognition timeout time “t_r” has elapsed with a speech (step S409). For example, the information processing system 1 determines whether or not a condition (hereinafter also referred to as “speech end determination condition” that the session timeout time “t_s” has elapsed without a speech or the voice recognition timeout time “t_r” has elapsed with a speech is satisfied.

Furthermore, the information processing system 1 performs the processing of step S410 in the case where the condition that the session timeout time “t_s” has elapsed without a speech or the voice recognition timeout time “t_r” has elapsed with a speech is not satisfied (step S409: No). The information processing system 1 performs the processing of step S410 in the case where the speech end determination condition is not satisfied. For example, the information processing system 1 executes the processing of step S410 in the case where the session timeout time “t_s” has not elapsed without a speech or the voice recognition timeout time “t_r” has not elapsed with a speech.

The information processing system 1 waits for the short span time “T_s” and waits for update of the respiratory state vector “H_v” (step S410). Thereafter, the information processing system 1 returns to step S405 and repeats the processing.

Furthermore, the information processing system 1 waits until the speed “v_b” becomes smaller than the threshold “v_f” (step S411) in the case where the speed “v_b” of the respiratory state vector “H_v” is not smaller than threshold “v_f” (step S406: No). The information processing system 1 waits until the speed “v_b” becomes smaller than the threshold “v_f” in the case where the speed “v_b” of the respiratory state vector “H_v” is equal to or larger than the threshold “v_f”. Thereafter, the information processing system 1 returns to step S401 in FIG. 12A and repeats the processing.

Furthermore, the information processing system 1 waits until the speed “v_b” becomes larger than the threshold “v_s” (step S412) in the case where the speed “v_b” of the respiratory state vector “H_v” is not larger than threshold “v_s” (step S408: No). The information processing system 1 waits until the speed “v_b” becomes larger than the threshold “v_s” in the case where the speed “v_b” of the respiratory state vector “H_v” is equal to or smaller than the threshold “v_s”. Thereafter, the information processing system 1 returns to step S401 in FIG. 12A and repeats the processing.

Furthermore, the information processing system 1 performs the processing of step S413 in FIG. 12A in the case where the condition that the session timeout time “t_s” has elapsed without a speech or the voice recognition timeout time “t_r” has elapsed with a speech is satisfied (step S409: Yes). The information processing system 1 performs the processing of step S413 in FIG. 12A in the case where the speech end determination condition is satisfied.

The information processing system 1 determines whether or not the user has not spoken within the session timeout time “t_s” (step S413). In a case where the user has not spoken within the time of the session timeout time “t_s” (step S413: No), the information processing system 1 terminates the processing.

On the other hand, in a case where the user has spoken within the session timeout time “t_s” (step S413: Yes), the information processing system 1 determines whether or not the result of intent understanding (Intent) of the user speech is interpretable (step S414). For example, the information processing system 1 determines whether or not the result of intent understanding (Intent) of the user speech is not OOD.

In the case where the result of intent understanding (Intent) of the user speech is interpretable (step S414: Yes), the information processing system 1 determines whether or not an interaction scenario of the voice interaction session has been completed (step S415). For example, in a case where the result of intent understanding (Intent) of the user speech is other than OOD, the information processing system 1 performs the processing of step S415.

In a case where the interaction scenario of the voice interaction session has been completed (step S415: Yes), the information processing system 1 terminates the processing.

On the other hand, in a case where the interaction scenario of the voice interaction session has not been completed (step S415: No), the information processing system 1 returns to step S401 and repeats the processing.

Furthermore, in the case where the result (Intent) of intent understanding of the user speech is not interpretable (step S414: No), the information processing system 1 determines whether or not the respiratory state vector “H_v” of the respiration falls outside the normal range “R_N” as illustrated in FIG. 12B (step S416). For example, in a case where the result of intent understanding (Intent) of the user speech is OOD, the information processing system 1 performs the processing of step S416 in FIG. 12B.

As illustrated in FIG. 12B, the information processing system 1 determines whether or not respiratory state vector “H_v” of the respiration falls outside the normal range “R_N” (step S416). The information processing system 1 saves the state of the voice interaction session (step S417) in the case where the respiratory state vector “H_v” of the respiration falls outside the normal range “R_N” (step S416: Yes). Then, the information processing system 1 interrupts the voice interaction session (step S418).

Then, the information processing system 1 determines whether or not the respiratory state vector “H_v” of the respiration does not fall within the normal range “R_N” (step S419). That is, the information processing system 1 determines whether or not respiratory state vector “H_v” of the respiration falls outside the normal range “R_N”.

The information processing system 1 repeats the processing of step S419 in the case where the respiratory state vector “H_v” of the respiration does not fall within the normal range “R_N” (step S419: No). For example, the information processing system 1 waits until the respiratory state vector “H_v” of the respiration falls within the normal range “R_N”, that is, the respiration of the user calms down in the case where the respiratory state vector “H_v” of the respiration does not fall within the normal range “R_N”.

The information processing system 1 resumes the voice interaction session from the saved state (step S420) in the case where the respiratory state vector “H_v” of the respiration falls within the normal range “R_N” (step S419: Yes). Then, the information processing system 1 executes the processing of step S413 in FIG. 12A.

On the other hand, in the case where the respiratory state vector “H_v” of the respiration does not fall outside the normal range “R_N” (step S416: No), the information processing system 1 performs the system speech of rehearing the speech in which the user's Intent is OOD (step S421). For example, in the case where the respiratory state vector “H_v” of the respiration falls within the normal range “R_N”, that is, in the case where the user's respiration is calm, the information processing system 1 performs the rehearing speech (for example, “Please say that again” or the like) with respect to the speech in which the user's speech intent is uninterpretable. Then, the information processing system 1 executes the processing of step S413.

[2-3-1. Effects and the Like]

For example, as examples of the respiratory state in which the voice speech becomes difficult due to out of breath by exercise include a case where the respiration becomes shallow due to tension, stress, concentration, or the like, a case of arrested respiration or hyperventilation, a case where the frequency of respiration decreases due to drowsiness, a case of respiratory physiological phenomena such as cough and sneeze, and a case where short-term respiration stops (becomes shallow) due to surprise or strain. The information processing system 1 enables a plurality of speeches of the user to be appropriately concatenated by the above-described processing even in such a case.

Furthermore, the information processing system 1 resumes the interaction session after the respiration is recovered to a normal voice recognition rate when the voice recognition rate due to out of breath is lowered (at the time of OOD speech) by the above-described processing. Therefore, the information processing system 1 can suppress unnecessary restatement such as when the user is out of breath and the voice recognition cannot be performed. Furthermore, the information processing system 1 exhibits effects other than out of breath due to exercise. For example, the information processing system 1 can obtain similar effects to those in the case of out of breath due to exercise even in a situation where it is difficult to speak due to physiological phenomena such as tension/stress, concentration/arrested respiration, hyperventilation, drowsiness, cough and sneeze, or surprise/strain by the voice interaction control using a generalized respiratory state vector.

2-4. Relationship Example Between Respiratory State and Voice Interaction Control

Here, a relationship between the respiratory state of the user and the voice interaction control of the information processing system 1 will be described with reference to FIG. 14. FIG. 14 is a diagram illustrating an example of a relationship between the respiratory state and the voice interaction control. For example, FIG. 14 illustrates an influence of the instantaneous speed of the respiration “v_b” on the voice speech and an interaction control method.

The table illustrated in FIG. 14 illustrates examples of state/behavior of the user affecting the speech and the interaction control, corresponding to the observed respiratory state including the depth “d_b”, the frequency “f_b”, the short-term speed “v_b” of the respiration of the user, and the like.

For example, in a case where the depth “d_b”, the frequency “f_b”, and the short-term speed “v_b” are all normal, the case indicates that the user's state/behavior is estimated to be able to make a normal speech in a calm state. Furthermore, as the interaction control in this case, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are controlled with priority given to response.

Furthermore, the case indicates that (normal) control is performed assuming that the processing after the OOD speech is not caused by the respiration. In a case where at least one of the depth “d_b”, the frequency “f_b”, or the short-term speed “v_b” is not normal, the case indicates that control to resume the interaction is performed when the respiratory state vector “H_v” falls within the normal range “R_N” for the processing after the OOD speech.

For example, in a case where the depth “d_b” is deep, the frequency “f_b” is high, and the short-term speed “v_b” is fast, the case indicates that the user's state/behavior is estimated to be out of breath or hyperventilation. Furthermore, as the interaction control in this case, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended in proportion to the voice speech influence level “E_u”.

For example, in a case where the depth “d_b” is deep or normal, the frequency “f_b” is high or normal, and the short-term speed “v_b” is (instantaneously) fast, the case indicates that the user's state/behavior is estimated to be a respiratory physiological phenomenon such as cough, sneezing, yawning, or sighing. Furthermore, as the interaction control in this case, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended in proportion to the voice speech influence level “E_u”. Furthermore, in this case, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended by a period in which the speed “v_b” is equal to or larger than the threshold “v_f”. For example, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended during a period in which the speed “v_b” is equal to or larger than the threshold “v_f”.

For example, in a case where the depth “d_b” is shallow, the frequency “f_b” is low, and the short-term speed “v_b” is slow, the case indicates that the user's state/behavior is estimated to be focused or arrested respiration. Furthermore, as the interaction control in this case, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended in proportion to the voice speech influence level “E_u”.

For example, in a case where the depth “d_b” is shallow or normal, the frequency “f_b” is low or normal, and the short-term speed “v_b” is (instantaneously) slow, it indicates that the user's state/behavior is estimated to hold his/her breath or allow breath to become shallow due to surprise or strain. Furthermore, as the interaction control in this case, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended in proportion to the voice speech influence level “E_u”. Furthermore, in this case, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended by a period in which the speed “v_b” is equal to or smaller than the threshold “v_s”. For example, the voice recognition timeout time “t_r” during the speech and the session timeout time “t_s” before the speech are extended during a period in which the speed “v_b” is equal to or smaller than the threshold “v_s”.

3. Configuration and Processing of Information Processing System

From here, the configuration and processing of the information processing system will be described below with reference to FIGS. 15 to 23B. Note that the points to be described below may be applied to either the information processing system 1 according to the first embodiment or the information processing system 1 according to the second embodiment.

3-1. Functional Configuration Example of Information Processing System

FIG. 15 is a diagram illustrating a functional configuration example of the information processing system. In FIG. 15, the left side of the broken line BS corresponds to components on the terminal device 10 side, and the right side of the broken line BS corresponds to components on the server device 100 side. The broken line BS indicates an example of allocation of functions between the terminal device 10 and the server device 100 in the information processing system 1. In FIG. 15, each component illustrated on the left side of the broken line BS is implemented by the terminal device 10. Furthermore, in FIG. 15, each component illustrated on the right side of the broken line BS in FIG. 15 is implemented by the server device 100. Note that a boundary (interface) of a device configuration in the information processing system 1 is not limited to the broken line BS, and the functions allocated to the terminal device 10 and the server device 100 may be any combination.

In the information processing system 1, a user's speech voice is input to the system through a voice input device such as a microphone, and a speech section is detected by voice activity detection (VAD). In the information processing system 1, a signal detected as a speech section by the VAD undergoes automatic speech recognition (ASR) processing and converted into a text. In the information processing system 1, a speech intent (Intent) and attribute information (Entity) to be a speech target of the user speech converted into the text are estimated by semantic understanding processing (NLU) and input to the voice interaction session control. In the information processing system 1, in a case where the speech intent is uninterpretable in NLU, Intent is input to the voice interaction session control as out of domain (OOD).

In the information processing system 1, the respiration of the user is observed as a displacement value of a respiration amount by the respiration sensor. In the information processing system 1, the respiratory state is detected from the observed displacement value of the respiration amount by the respiratory state detection, and is input to the voice interaction session control. In the information processing system 1 of FIG. 15, a degree of roughness “H_b” and the respiratory state vector “H_v” are input to the voice interaction session control.

In the information processing system 1, a user speech text from ASR is also input to the voice interaction session control. In the voice interaction session control in the information processing system 1, a plurality of user speech texts is concatenated according to the respiratory state, and is input to NLU as a concatenated speech text. In the voice interaction session control in the information processing system 1 of FIG. 15, a plurality of user speech texts is concatenated according to the degree of roughness “H_b” and the respiratory state vector “H_v”, and is input to NLU as the concatenated speech text.

In NLU in the information processing system 1, Intent and Entity are estimated for the concatenated speech text input from the voice interaction session control in addition to a user speech text input from ASR, and Intent and Entity are input to the voice interaction session control.

In the voice interaction session control in the information processing system 1, the silent timeout time of the voice recognition speech end determination and the voice interaction session end determination, and the interruption/resumption of the voice interaction session are controlled on the basis of the input respiratory state of the user and Intent and Entity of the speech. In the information processing system 1, response generation generates a system speech text in accordance with an instruction from voice interaction session control. In the information processing system 1, the system speech text undergoes voice synthesis processing and synthesized into a system speech voice signal, and then spoken to the user by voice through an output device such as a speaker. Note that the above is an example, and the information processing system 1 may implement each function by various configurations.

3-2. Respiratory State Vector

Next, points related to processing using the respiratory state vector will be described with reference to FIGS. 16 to 21C.

[3-2-1. Example of Observation Target Time in Respiratory State Vector Detection]

FIG. 16 is a diagram illustrating an example of an observation target time in respiratory state vector detection. FIG. 16 illustrates an observation target time in the respiratory state vector detection. In FIG. 16, calculation of four respiratory state vectors “H_v” is illustrated as H_v calculation #1 to #4. The bar corresponding to each of the H_v calculations #1 to #4 indicates a sample to be observed corresponding to each calculation in an abstract manner.

As illustrated in FIG. 16, each of the H_v calculation #1 to #4 is continuously performed while being shifted by the short span time T_s. That is, the information processing system 1 repeats the calculation of the respiratory state vector “H_v” in the cycle of the short span time T_s. The long span time T₁, “n₁”, and “n_s” in FIG. 16 are similar to those described above, and thus description thereof is omitted.

As illustrated in FIG. 16, the respiratory state vector “H_v” is calculated for each short span time T_s (the number of observation samples n_s). In this manner, the number of observation samples n₁ of the long span time T₁ is calculated in an overlapping manner for n₁−n_s samples.

[3-2-2. Example of Observation Value in Respiratory State Vector Detection]

FIG. 17 is a diagram illustrating an example of observation values in the respiratory state vector detection. A graph GR1 in FIG. 17 illustrates an example of various observation values in a certain long span time T₁. In FIG. 17, the average observation value “Sm” is illustrated by a solid line extending in a horizontal direction within the long span time T₁.

Furthermore, each of observed peak values “S_p1” to “S_p7” indicates a peak observation value of a maximum value or a minimum value between intersections and “S_m”. An arrow extending in a vertical direction from the average observation value “Sm” toward the maximum value or the minimum value of the waveform indicates a term of “S_pi−S_m” that is a target for calculating RMS by the above equation (9). Furthermore, in FIG. 17, since the number of observed peak values is seven observed peak values “S_p1” to “S_p7”, the number of peaks “N_1p” is 7. For example, the information processing system 1 calculates an average of peak absolute values in the n₁ sample in the most recent long span time T₁ as the depth of the respiration. The information processing system 1 calculates the respiratory state vector “H_v” using such respiration information of the user.

[3-2-3. Example of Normal Range by Long Span Time Observation Element]

FIG. 18 is a diagram illustrating an example of the normal range by long span time observation elements. FIG. 18 illustrates a case where a right-left direction (horizontal direction) is an axis corresponding to the depth “d_b” and tan up-down direction (vertical direction) is an axis corresponding to the frequency “f_b”.

Note that a depth direction of the paper surface of FIG. 18 is an axis corresponding to the speed “v_b”. That is, FIG. 18 illustrates a cross-sectional view in a case where a three-dimensional space having the depth “d_b”, the frequency “f_b”, and the speed “v_b” as axes is viewed from an axial direction (depth direction) corresponding to the speed “v_b”. For example, FIG. 18 illustrates a cross section at a position where the speed “v_b” is speed “v₀”. FIG. 18 illustrates an example of definition of the normal range “R_N” by the long span time observation elements (the depth “d_b” and the frequency “f_b”) of the respiratory state vector “H_v”.

A central portion in FIG. 18 is a range corresponding to the normal range “R_N”. The normal range “R_N” corresponds to the normal state of the user. For example, the normal range “R_N” corresponds to the state in which the user can speak as usual. Furthermore, in a case where the depth “d_b” and the frequency “f_b” are large, the case indicates that the user is exercising or in a hyperventilated state, the user cannot speak due to out of breath, the speak is interrupted, and the like. Furthermore, in a case where the depth “d_b” and the frequency “f_b” are small, the case indicates that the user is in a state of tension or stress, the voice becomes small, the voice tends to be muffled, it is difficult to hear what the user is saying, and the like.

Furthermore, in a case where the depth “d_b” and the frequency “f_b” are small, the case indicates that the user is in a state of concentration or arrested respiration, attention is not directed, the power of concentration itself is reduced, and the like. Furthermore, in a case where the depth “d_b” is large and the frequency “f_b” is small, the case indicates a drowsiness or sleep state, which is a suitable state for speech, but if it goes too far, it becomes difficult to speak due to drowsiness.

[3-2-4. Example of Relationship Between Each Element of Respiratory State Vector and Respiratory State of User]

Next, examples of a relationship between each element of the respiratory state vector and the respiratory state of the user will be described with reference to FIGS. 19A to 21C. For example, FIGS. 19A to 21C illustrate points related to short span time observation elements.

[3-2-4-1. Example of Relationship With Speed]

First, a relationship with speed will be described with reference to FIGS. 19A to 19C. FIGS. 19A to 19C are diagrams illustrating examples of the relationship between each element of the respiratory state vector and the respiratory state of the user. FIGS. 19A to 19C illustrate cross sections orthogonal to the speed “v_b” (the depth of the paper surface) direction in FIG. 18.

FIG. 19A illustrates definition of the normal range “R_N” by the short span time observation element (the speed “v_b”) of the respiratory state vector “H_v” in the case where the speed “v_b” is slow. That is, FIG. 19A illustrates a cross section on a front side of the paper surface of FIG. 18. The cross section illustrated in FIG. 19A includes an area (for example, the first, second, fourth quadrants, and the like) indicating that the user has held his/her breath due to surprise, strain, or the like or the breath has become shallow, and an area (for example, the third quadrant and the like) indistinguishable from the state of concentration or arrested respiration.

The information processing system 1 can estimate that the user has held his/her breath or the breath has become shallow due to surprise, strength, or the like when the instantaneous respiration speed becomes slow (becomes equal to or less than the threshold “v_s”). Note that the range of f_b0 and d_b0 is an area indistinguishable from concentration/arrested respiration.

FIG. 19B illustrates definition of the normal range “R_N” by the short span time observation element (the speed “v_b”) of the respiratory state vector “H_v” in the case where the speed “v_b” is normal. That is, FIG. 19B illustrates a cross section on the paper surface of FIG. 18. In the cross section illustrated in FIG. 19B, the same similarly applies to FIG. 18 when the instantaneous respiration speed is normal.

FIG. 19C illustrates definition of the normal range “R_N” by the short span time observation element (the speed “v_b”) of the respiratory state vector “H_v” in the case where the speed “v_b” is fast. That is, FIG. 19C illustrates a cross section on a depth side of the paper surface of FIG. 18. The cross section illustrated in FIG. 19C includes an area (for example, the second, third, fourth quadrants, and the like) indicating a physiological phenomenon of the respiratory organ such as cough, sneezing, or hiccup, and an area (for example, the first quadrant and the like) indistinguishable from the state during exercise or hyperventilation.

The information processing system 1 can estimate that it is a physiological phenomenon such as cough, sneezing, hiccup, yawning, or sighing when the instantaneous respiration speed increases (when the instantaneous respiration speed becomes equal to or larger than the threshold “v_f”). Note that the range of f_b>f₀ and d_b>d₀ is an area indistinguishable from during exercise and hyperventilation.

[3-2-4-2. Example of Relationship With Frequency]

Second, a relationship with frequency will be described with reference to FIGS. 20A to 20C. FIGS. 20A to 20C are diagrams illustrating an example of a relationship between each element of a respiratory state vector and a respiratory state of a user. FIGS. 20A to 20C illustrate cross sections orthogonal to the frequency “f_b” direction in FIG. 18. That is, FIGS. 20A to 20C illustrate cross sections in a case where FIG. 18 is viewed from the vertical direction (up-down direction).

FIG. 20A illustrates definition of the normal range “R_N” by the short span time observation element (the frequency “f_b”) of the respiratory state vector “H_v” in the case where the frequency “f_b” is low. That is, FIG. 20A illustrates a cross section at a position where the frequency is smaller than the frequency “f₀” of the axis of the frequency “f_b”.

The information processing system 1 can estimate that it is a physiological phenomenon such as cough, sneezing, hiccup, yawning, or sighing when the instantaneous respiration speed becomes v_b≥v_f in the cross section illustrated in FIG. 20A. Furthermore, in the cross section illustrated in FIG. 20A, the information processing system 1 can estimate that the user has held his/her breath due to surprise, strain, or the like or the breath has become shallow when v_b≤v_s is satisfied except for the direction of concentration/arrested respiration (d_b0).

FIG. 20B illustrates definition of the normal range “R_N” by the short span time observation element (the frequency “f_b”) of the respiratory state vector “H_v” in the case where the frequency “f_b” is normal. That is, FIG. 20A illustrates a cross section at a position where the frequency “f_b” is the frequency “f₀”.

The information processing system 1 can estimate that it is a physiological phenomenon such as cough, sneezing, hiccup, yawning, or sighing when the instantaneous respiration speed becomes v_b≥v_f in the cross section illustrated in FIG. 20B. Furthermore, in the cross section illustrated in FIG. 20B, the information processing system 1 can estimate that the user has held his/her breath due to surprise, strain, or the like or the breath has become shallow when v_b≤v_s is satisfied.

FIG. 20C illustrates definition of the normal range “R_N” by the short span time observation element (the frequency “f_b”) of the respiratory state vector “H_v” in the case where the frequency “f_b” is high. That is, FIG. 20C illustrates a cross section at a position where the frequency is larger than the frequency “f₀” of the axis of the frequency “f_b”.

The information processing system 1 can estimate that it is a physiological phenomenon such as cough, sneezing, hiccup, yawning, or sighing when the instantaneous respiration speed becomes v_b≥v_f except for the direction of exercise and hyperventilation (d_b>d₀) in the cross section illustrated in FIG. 20C. Furthermore, in the cross section illustrated in FIG. 20C, the information processing system 1 can estimate that the user has held his/her breath due to surprise, strain, or the like or the breath has become shallow when v_b≤v_s is satisfied.

[3-2-4-3. Example of Relationship With Depth]

Next, a relationship with depth will be described with reference to FIGS. 21A to 21C. FIGS. 21A to 21C are diagrams illustrating an example of a relationship between each element of a respiratory state vector and a respiratory state of a user. FIGS. 21A to 21C illustrate cross sections orthogonal to the depth “d_b” direction of FIG. 18. That is, FIGS. 21A to 21C illustrate cross sections in a case where FIG. 18 is viewed from the horizontal direction (right-left direction).

FIG. 21A illustrates definition of the normal range “R_N” by the short span time observation element (the depth “d_b”) of the respiratory state vector “H_v” in the case where the depth “d_b” is shallow. That is, FIG. 21A illustrates a cross section at a position where the depth is smaller than the depth “d₀” of the axis of the depth “d_b”.

The information processing system 1 can estimate that it is a physiological phenomenon such as cough, sneezing, hiccup, yawning, or sighing when the instantaneous respiration speed becomes v_b≥v_f in the cross section illustrated in FIG. 21A. Furthermore, in the cross section illustrated in FIG. 21A, the information processing system 1 can estimate that the user has held his/her breath due to surprise, strain, or the like or the breath has become shallow when v_b≤v_s is satisfied except for the direction of concentration/arrested respiration (f_b0).

FIG. 21B illustrates definition of the normal range “R_N” by the short span time observation element (the depth “d_b”) of the respiratory state vector “H_v” in the case where the depth “d_b” is normal. That is, FIG. 21A illustrates a cross section at a position where the depth “d_b” is the depth “d₀”.

The information processing system 1 can estimate that it is a physiological phenomenon such as cough, sneezing, hiccup, yawning, or sighing when the instantaneous respiration speed becomes v_b≥v_f in the cross section illustrated in FIG. 21B. Furthermore, in the cross section illustrated in FIG. 21B, the information processing system 1 can estimate that the user has held his/her breath due to surprise, strain, or the like or the breath has become shallow when v_b≤v_s is satisfied.

FIG. 21C illustrates definition of the normal range “R_N” by the short span time observation element (the depth “d_b”) of the respiratory state vector “H_v” in the case where the depth “d_b” is deep. That is, FIG. 21C illustrates a cross section at a position where the depth is larger than the depth “d₀” of the axis of the depth “d_b”.

The information processing system 1 can estimate that it is a physiological phenomenon such as cough, sneezing, hiccup, yawning, or sighing when the instantaneous respiration speed becomes v_b≥v_f except for the direction of exercise and hyperventilation (f_b>f₀) in the cross section illustrated in FIG. 21C. Furthermore, in the cross section illustrated in FIG. 21C, the information processing system 1 can estimate that the user has held his/her breath due to surprise, strain, or the like or the breath has become shallow when v_b≤v_s is satisfied.

3-3. Processing Example 1

Next, an example of processing will be described with reference to FIGS. 22A and 22B. Note that, in FIGS. 22A and 22B, either the degree of roughness “H_b” or the respiratory state vector “H_v” may be used to estimate the respiratory state of the user. Note that, in FIGS. 22A and 22B, description of points similar to those in FIGS. 1 and 9 is omitted.

[3-3-1. Processing Example in Normal Times]

First, an example of processing in normal times will be described with reference to FIG. 22A. FIG. 22A is a diagram illustrating an example of processing in normal times. For example, FIG. 22A illustrates an example of a case where the degree of roughness “H_b” is less than the specified threshold “H_th” or the respiratory state vector “H_v” is within the normal range “R_N”. That is, FIG. 22A illustrates a case where the silent timeout times “t_r” and “t_s” are not extended.

In FIG. 22A, the information processing system 1 performs a system output of “A message has arrived from Mr. oo. Shall I read out?”. In response to the system output, the user U1 speaks “Read out” before the silent timeout time “t_s” elapses, which is the interaction session timeout time, after the end of the system output. Then, after the voice recognition timeout time “t_r” has elapsed, the information processing system 1 executes the processing such as the voice recognition. The information processing system 1 recognizes (estimates) Intent indicating the speech intent of the user U1 as “ReadOut”.

Then, the information processing system 1 outputs the message of Mr. oo to the user U1 according to the result of the voice recognition. In FIG. 22A, the information processing system 1 outputs the message of Mr. oo “Can you come here right now?” to the user U1 by voice. Then, the information processing system 1 makes a speech of “Would you like to reply?”.

In response to the system output, the user U1 speaks “reply” before the silent timeout time “t_s” elapses, which is the interaction session timeout time, after the end of the system output. Then, after the voice recognition timeout time “t_r” has elapsed, the information processing system 1 executes the processing such as the voice recognition. The information processing system 1 recognizes (estimates) Intent indicating speech intent of the user U1 as “Reply”.

Then, the information processing system 1 makes a speech according to the result of the voice recognition. In FIG. 22A, the information processing system 1 makes a speech of “Give me a reply message, please”.

In response to the system output, the user U1 makes a speech of “It's not possible right now” before the silent timeout time “t_s” elapses, which is the interaction session timeout time, after the end of the system output. Then, after the voice recognition timeout time “t_r” has elapsed, the information processing system 1 determines (estimates) “Dictation End”. For example, the information processing system 1 transmits textual information of “It's not possible right now” to the terminal device 10 of Mr. oo.

Then, the information processing system 1 makes a speech in accordance with the processing. In FIG. 22A, the information processing system 1 makes a speech of “replied”.

As described above, in the information processing system 1 does not extend the timeout time of the voice recognition and the interaction session when the user's respiration is normal. Therefore, the information processing system 1 can perform a response speech without causing an unnecessary waiting time after the user speech. Thereby, the information processing system 1 can provide a service without impairing existing interaction response performance in normal times.

[3-3-2. Processing Example During Exercise]

Next, an example of processing during exercise will be described with reference to FIG. 22B. FIG. 22B is a diagram illustrating an example of processing during exercise. For example, FIG. 22B is a diagram illustrating an example of processing while the user U1 is moving (exercising) by pedaling a bicycle. For example, FIG. 22B illustrates an example of a case where the degree of roughness “H_b” is equal to or larger than the specified threshold “H_th” or the respiratory state vector “H_v” falls outside the normal range “R_N”. That is, FIG. 22B illustrates a case where the silent timeout times “t_r” and “t_s” are extended.

In FIG. 22B, the information processing system 1 performs a system output of “A message has arrived from Mr. oo. Shall I read out?”. In response to the system output, the user U1 speaks “read” before the extended silent timeout time “t_s” elapses after the end of the system output, and speaks “out” before the extended silent timeout time “t_r” elapses. In this case, the speech of “read” corresponds to the first speech, and the speech of “out” corresponds to the second speech.

Here, the information processing system 1 extends the silent timeout time “t_r”, which is the voice recognition timeout time, to be longer as the voice speech influence level “E_u” is larger. Thereby, even in the user's speech that is interrupted due to out of breath, the silent timeout time “t_r” that is the voice recognition timeout time is extended by the time according to the value of the voice speech influence level “E_u”. Therefore, the information processing system 1 can accept even a plurality of discontinuous and intermittent speeches as one speech.

Then, after the voice recognition timeout time “t_r” has elapsed, the information processing system 1 executes the processing such as the voice recognition. The information processing system 1 executes the processing such as the voice recognition using “read out” in which the speech of “read” of the user U1 and the speech of “up” of the user U1 are concatenated as one speech. Then, the information processing system 1 recognizes (estimates) Intent indicating speech intent of the user U1 as “ReadOut”.

Then, the information processing system 1 outputs the message of Mr. oo to the user U1 according to the result of the voice recognition. In FIG. 22B, the information processing system 1 outputs a message of Mr. oo to the user U1 “Can you come here right now?” by voice. Then, the information processing system 1 makes a speech of “Would you like to reply?”.

Meanwhile, in FIG. 22B, since the user U1 is exercising and is out of breath, the user cannot respond for a while.

Here, the information processing system 1 extends the silent timeout time “t_s”, which is the interaction session timeout time, to be longer as the voice speech influence level “E_u” is larger. Thereby, even in the case where the user cannot start a speech as desired due to out of breath, the silent timeout time “t_s” that is the interaction session timeout time is extended by the time according to the value of the voice speech influence level “E_u”. Therefore, the information processing system 1 can suppress the end of the interactive session in the case where the user cannot start the speech as desired due to out of breath, and can accept the user's speech in the session.

In FIG. 22B, the user U1 speaks “reply” before the extended silent timeout time “t_s” elapses after the end of the system output, and speaks “please” before the extended silent timeout time “t_r” elapses. In this case, the speech of “reply” corresponds to the first speech, and the speech of “please” corresponds to the second speech.

Then, after the voice recognition timeout time “t_r” has elapsed, the information processing system 1 executes the processing such as the voice recognition. The information processing system 1 executes the processing such as the voice recognition using “reply please” in which the speech of “reply” of the user U1 and the speech of “please” of the user U1 are concatenated as one speech. The information processing system 1 recognizes (estimates) Intent indicating speech intent of the user U1 as “Reply”.

Then, the information processing system 1 makes a speech according to the result of the voice recognition. In FIG. 22B, the information processing system 1 makes a speech of “Give me a reply message, please”.

In response to the system output, the user U1 speaks “It's” before the silent timeout time “t_s” elapses, which is the interaction session timeout time, after the end of the system output, and speaks “not possible” before the extended silent timeout time “t_r” elapses. Then, the user U1 speaks “right” before the extended silent timeout time “t_r” elapses, and speaks “now” before the extended silent timeout time “t_r” elapses. In this case, the speech “It's” corresponds to the first speech, the speech “not possible” corresponds to the second speech, the speech “right” corresponds to the third speech, and the speech “now” corresponds to the fourth speech. Note that the speech “right” is the second speech for the speech “not possible”. Furthermore, “now” is the third speech for the speech “not possible”, and is a second speech for the speech “right”.

Then, after the voice recognition timeout time “t_r” has elapsed, the information processing system 1 determines (estimates) “Dictation End”. The information processing system 1 transmits, to the terminal device 10 of Mr. oo, textual information of “It's not possible right now” in which the speech of “Its”, the speech of “not possible”, the speech of “right”, and the speech of “now” by the user U1 are concatenated into one speech.

As described above, since the voice recognition timeout time “t_r” is similarly extended at the time of dictating a free speech (Dictation), the information processing system 1 can prevent the message input from being terminated at an unintended position in the middle of the message input due to the interrupted speech.

Then, the information processing system 1 makes a speech in accordance with the processing. For example, the information processing system 1 makes a speech of “replied”.

3-4. Processing Example 2

Next, an example of another processing will be described with reference to FIGS. 23A and 23B. Note that, in FIGS. 23A and 23B, either the degree of roughness “H_b” or the respiratory state vector “H_v” may be used to estimate the respiratory state of the user. For example, FIGS. 23A and 23B are diagrams illustrating an example of processing while the user U1 is moving by riding a bicycle. Note that, in FIGS. 23A and 23B, description of points similar to those in FIGS. 1 and 9 is omitted.

[3-4-1. Processing Example During Exercise]

First, an example of processing in normal times will be described with reference to FIG. 23A. FIG. 23A is a diagram illustrating an example of processing during exercise. For example, FIG. 23A illustrates an example of a case where the degree of roughness “H_b” is equal to or larger than the specified threshold “H_th” or the respiratory state vector “H_v” falls outside the normal range “R_N”. That is, FIG. 23A illustrates a case where the silent timeout times “t_r” and “t_s” are extended.

In FIG. 23A, the information processing system 1 performs a system output of “A message has arrived from Mr. oo. Shall I read out?”. In response to the system output, the user U1 speaks “read” before the extended silent timeout time “t_s” elapses after the end of the system output, and speaks “out” before the extended silent timeout time “t_r” elapses. In this case, the speech of “read” corresponds to the first speech, and the speech of “out” corresponds to the second speech. FIG. 23A illustrates a case where the user U1 is out of breath during exercise, and the information processing system 1 cannot recognize the speech of the user U1.

After the voice recognition timeout time “t_r” has elapsed, the information processing system 1 executes the processing such as the voice recognition. As described above, since the information processing system 1 has not been able to recognize the speech of the user U1, the information processing system 1 recognizes (estimates) Intent indicating the speech intent of the user U1 as “OOD”. In other words, since the information processing system 1 has not been able to recognize the speech of the user U1, the information processing system 1 determines that the speech of the user U1 is uninterpretable.

Then, the information processing system 1 outputs the message of Mr. oo to the user U1 according to the result of the voice recognition. The information processing system 1 estimates that the degree of roughness “H_b” is equal to or larger than the specified threshold “H_th” or the respiratory state vector “H_v” is outside the normal range “R_N”, and the state of the user is other than the normal time, and notifies the user U1 that the message will be notified again. In FIG. 23A, the information processing system 1 makes a speech “The message will be notified later again”.

As described above, in the case where the user's respiration is rough, the speech is disturbed, and the voice recognition cannot be correctly performed, and the semantic understanding result is recognized (estimated) as OOD, the information processing system 1 saves the interaction state and temporarily interrupts the voice interaction session. There is a case where voice recognition cannot be correctly performed depending on the user's speech phrase when the respiration is rough, and in this case, there is a low possibility that voice recognition is correctly performed even if restatement is performed. Therefore, the information processing system 1 waits until the respiratory state of the user returns to a state that does not disturb the speech.

[3-4-2. Processing Example After Calming Down]

Next, an example of processing after calming down after exercise will be described with reference FIG. 23B. FIG. 23B is a diagram illustrating an example of processing after returning to normal times from during exercise. Specifically, FIG. 23B is a diagram illustrating an example of processing in a case where the state returns to the normal times of the user after elapse of time from the exercise of the user in FIG. 23A. For example, FIG. 23B illustrates an example of a case where the degree of roughness “H_b” is less than the specified threshold “H_th” or the respiratory state vector “H_v” is within the normal range “R_N”. That is, FIG. 23B illustrates the case where the silent timeout times “t_r” and “t_s” are not extended.

In FIG. 23B, since the degree of roughness “H_b” indicating the respiratory state of the user U1 is less than the specified threshold “H_th”, or the respiratory state vector “H_v” is within the normal range “R_N”, the information processing system 1 performs the notification in FIG. 23A again. The information processing system 1 performs the system output of “A message has arrived from the said Mr. oo. Shall I read out?”. In response to the system output, the user U1 speaks “Read out” before the silent timeout time “t_s” elapses, which is the interaction session timeout time, after the end of the system output. Then, after the voice recognition timeout time “t_r” has elapsed, the information processing system 1 executes the processing such as the voice recognition. The information processing system 1 recognizes (estimates) Intent indicating the speech intent of the user U1 as “ReadOut”.

Then, the information processing system 1 outputs the message of Mr. oo to the user U1 according to the result of the voice recognition. In FIG. 23B, the information processing system 1 outputs the message of Mr. oo to the user U1 “Can you come here right now?” by voice. Then, the information processing system 1 makes a speech of “Would you like to reply?”. Then, as illustrated in FIG. 22A, the information processing system 1 interacts with the user according to the response of the user, and provides a service according to the request of the user.

As described above, when the respiratory state of the user calms down and either the degree of roughness “H_b” being less than the specified threshold “H_th” or the respiratory state vector “H_v” falling within the normal range “R_N” is detected, the information processing system 1 resumes the voice interaction session from the saved interaction state. As a result, since the information processing system 1 can notify the user after the user becomes calm, the information processing system can appropriately provide a service.

4. Other Embodiments

The processing according to each of the above-described embodiments may be performed in various different forms (modifications) in addition to the above-described embodiments and modifications.

4-1. Modification

Hereinafter, each of modifications will be described.

[4-1-1. Example of Case Where Attention of User Is Distracted By Others]

First, an example of a case where attention of the user is distracted by others will be described. Specifically, an adaptation example of the interaction control by the respiratory state vector “H_v” in a use case where the attention of the user is distracted by others than the interaction with the system (information processing system 1) will be described.

For example, at the moment when the user's attention is distracted by others, the information processing system 1 performs processing as follows. In this case, since the instantaneous respiration speed “v_b” is equal to or less than the specified value “v_s” (threshold “v_s”) (correlated with “surprise/strain” that makes the breath shallow), the information processing system 1 extends the silent timeout times “t_r” and “t_s” until the speed “v_b” returns to a value larger than the specified value “v_s”.

Since the speed “v_b” returns to the value larger than the specified value “v_s” in a case where the attention is restored to the interaction with the system in a short time, the information processing system 1 cancels the extension of the silent timeout times “t_r” and “t_s” caused by the speed “v_b” in a short time.

For example, in a case where attention of the user is continuously distracted, the information processing system 1 performs processing as follows. Since the respiratory state vector “H_v” falls outside the normal range “R_N” (correlated with “concentration/tension” that makes the breath shallow) due to the depth “d_b”, the information processing system 1 extends the silent timeout times “t_r” and “t_s” according to the voice speech influence level “E_u”. In a case where there is no speech of the user even if the silent timeout time “t_s” has been extended so far, the information processing system 1 determines that the user has lost the intention of interaction with the system and thus times out and terminates the voice interaction session. Furthermore, in a case where the attention has been distracted by others and the user's speech to the system has been continuously interrupted even after the silent timeout time “t_r” has elapsed, the speech is interrupted halfway, resulting in the OOD speech, and the information processing system 1 interrupts the voice interaction session. The information processing system 1 resumes the voice interaction session when the respiratory state vector “H_v” falls within the normal range “R_N”.

For example, in a case where a speech other than the interaction with the system is made, the information processing system 1 performs processing as follows. The speech to the target that has distracted the attention becomes an OOD speech, and the information processing system 1 interrupts the interaction session. The information processing system 1 waits until the respiratory state vector “H_v” falls within the normal range “R_N” (attention returns to the interaction with the system) and then resumes the voice interaction session.

[4-1-2. Application Example to System Speech]

Next, an application example to the system speech will be described.

When the degree of roughness “H_b” of the respiration is equal to or larger than the specified threshold “H_th” or the respiratory state vector “H_v” is out of the normal range “R_N”, the information processing system 1 may decrease the speed of system speech by text-to-speech (TTS), increase the volume, or increase the pitch as the value of the voice speech influence level “E_u” increases. As described above, the information processing system 1 estimates that not only the user's ability of speaking but also a cognitive ability to listen and understand is deteriorated outside the normal range “R_N”, and the information processing system changes a system-side speech mode by decreasing the speed of the system speech, increasing the volume, or increasing the pitch.

[4-1-3. Modification by Personalized Learning]

Next, a modification by personalized learning will be described.

For example, the information processing system 1 stores, as learning information, a set of the respiratory state vectors “H_v” (labels within the normal range) when the interaction is smoothly established, and a set of the respiratory state vectors “H_v” (labels outside the normal range) when the silent timeout times “t_r” and “t_s” time out or an OOD speech occurs. For example, the server device 100A stores the learning information in the storage unit 120. Then, the information processing system 1 may perform the normal range “R_N” determination for the respiratory state vector “H_v” by performing class identification through machine learning using the learning information.

Furthermore, for the normal respiration origin “O_N”=(d₀, f₀, v₀), a preset initial value may be set on the basis of the depth, frequency, and speed at the time of general normal respiration, and the information processing system 1 may update the initial value with a value at which the likelihood of the normal range “R_N” is maximized by the class identification among values around the initial value. For example, the information processing system 1 may assign (set) values of the depth “d_b”, the frequency “f_b”, and the speed “v_b” around the initial value, apply the values to a class identifier generated in the machine learning, and update the normal respiration origin “O_N” with a combination of the depth “d_b”, the frequency “f_b”, and the speed “v_b” that maximizes the likelihood of the normal range “R_N”.

For example, the information processing system 1 stores a specific phrase P (the phrase itself is acquired from the speech after the respiratory state vector “H_v” falls within the normal range “R_N”) that has become the OOD speech and the respiratory state vector “H_vp” at this time in association with each other. For example, server device 100A stores information in which the specific phrase P and the respiratory state vector “H_vp” are associated with each other in the storage unit 120.

For example, a case where the information processing system 1 performs a system notification in which the user is highly likely to make a response speech with the stored specific phrase P will be described. In this case, in a case where the voice speech influence level “E_u” calculated from the current respiratory state vector “H_v” is equal to or larger than the voice speech influence level “E_u” calculated from the respiratory state vector “H_vp” stored in association with the specific phrase P, the information processing system 1 may wait and delay the notification itself until the respiratory state vector “H_v” falls within the normal range “R_N”. Furthermore, when performing similar system notification, the information processing system 1 may further extend the silent timeout times “t_r” and “t_s” from the time of the previous OOD speech.

With such personalized learning, the information processing system 1 can perform the interaction control optimized and adapted to (the difference in the influence of the respiration on the speech of) the user individual as the user further uses the system including the device and the like. Thereby, the information processing system 1 can absorb the difference in the influence of the respiration on the speech depending on an individual vital capacity or the like by personalized learning.

[4-1-4. Other Modifications]

Next, other modifications will be described.

For example, in a case of a device equipped with an image display device for the user, the information processing system 1 performs processing as follows. In a case where the image display device for the user is mounted as in the terminal device 10 having the display unit 16, the information processing system 1 performs processing as follows.

The information processing system 1 displays, with an indicator, the degree of roughness “H_b” of the respiration or the voice speech influence level “E_u” calculated from the respiratory state vector “H_v”. As described above, the information processing system 1 may feed back system behavior reasons such as the extension of the silent timeout times “t_r” and “t_s” and the interruption/resumption of the interaction to the user. Furthermore, the information processing system 1 may present the time until the timeout of the silent timeout times “t_r” and “t_s” by a countdown display or an indicator.

For example, in a case where an importance level of the system notification is set, the information processing system 1 performs processing as follows.

In the case of the notification with high importance, the information processing system 1 may store the notification when the extended silent timeout time “t_s” times out and the voice interaction session ends, and may perform re-notification after the respiration becomes normal. Furthermore, when the voice speech influence level “E_u” is higher than the specified value, the information processing system 1 may modify the system speech so that the user can respond with a simple speech such as Yes or No.

For example, in a case of a device equipped with a line-of-sight detection device, the information processing system 1 performs processing as follows. For example, in a case where the terminal device 10 used by the user has a line-of-sight detection function, the information processing system 1 performs processing as follows. In this case, the information processing system 1 may extend the silent timeout times “t_r” and “t_s” when the user is not looking at the voice interaction device by the line-of-sight detection.

4-2. Configuration Example of Performing Processing of Voice Interaction Control on Client Side

In the embodiment, as an example of the system configuration, the case where the server device 100 or 100A performs the processing of the voice interaction control and the like has been described, but the terminal device 10 may perform the processing of the voice interaction control. That is, the terminal device 10, which is a client-side device, may be an information processing device that performs the above-described processing of the voice interaction control. As described above, the system configuration of the information processing system 1 is not limited to the configuration in which the server devices 100 or 100A, which is the server-side device, performs the processing of the voice interaction control, and may be a configuration in which the terminal device 10, which is the client-side device, performs the above-described processing of the voice interaction control.

In the case where the terminal device 10 is an information processing device that performs the above-described processing of the voice interaction control, the processing of the voice interaction control is performed on the client side (terminal device 10) in the information processing system 1. Then, the server side (server device 100 or 100A) acquires various types of information from the terminal device 10 and performs various types of processing. In this case, the execution unit 152 of the terminal device 10 may have a function similar to that of the execution unit 134 of the server device 100 or 100A. Furthermore, the terminal device 10 may include a calculation unit that implements a function similar to that of the above-described calculation unit 132 and a determination unit that implements a function similar to that of the above-described determination unit 133. Furthermore, in this case, the server devices 100 or 100A may not include the calculation unit 132 or 132A and the determination unit 133 or 133A.

Furthermore, the information processing system 1 may have a system configuration in which the degree of roughness “H_b” that is a scalar value or the respiratory state vector “H_v” that is a vector is calculated on the client side (terminal device 10), and the processing of the voice interaction control is performed using the information of the degree of roughness “H_b” or the respiratory state vector “H_v” on the server side (server device 100 or 100A) that has received the information of the calculated degree of roughness “H_b” or respiratory state vector “H_v” from the client side. In this case, the terminal device 10 that is the client-side device may be an information processing device that performs the calculation processing of the above-described degree of roughness “H_b” and the respiratory state vector “H_v”, and the server device 100 or 100A that is the server-side device may be an information processing device that performs the processing of the voice interaction control using the above-described degree of roughness “H_b” and respiratory state vector “H_v”. In this case, for example, the calculation unit of the terminal device 10 performs the calculation processing, and the execution unit 134 of the server device 100 or 100A performs the processing of the voice interaction control.

Note that the above is an example, and any device may perform each processing in the information processing system 1. As described above, the information processing system 1 may have a system configuration in which either the client-side device (the terminal device 10) or the server-side device (the server device 100 or 100A) performs each processing.

4-3. Other Configuration Examples

In the above examples, the server devices 100 and 100A, and the terminal device 10 are separated from each other, but these devices may be integrated. Furthermore, the server device (information processing device) may perform the processing such as the voice interaction control using both the degree of roughness “H_b” and the respiratory state vector “H_v”. In this case, the server device may be an information processing device having functions of both the server device 100 and the server device 100A.

4-4. Others

Furthermore, among the pieces of processing described in the above embodiments, the entire or part of the processing described as being automatically performed can be manually performed, or the entire or part of the processing described as being manually performed can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters illustrated in the document and the drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing are not limited to the illustrated information.

Furthermore, each component of each device illustrated in the drawings is functionally conceptual, and is not necessarily physically configured as illustrated in the drawings. That is, a specific form of distribution and integration of each device is not limited to the illustrated form, and some or part thereof can be functionally or physically distributed and integrated in an arbitrary unit according to various loads, use conditions, and the like.

Furthermore, the above-described embodiments and modifications can be appropriately combined within a range in which the processing contents do not contradict each other.

Furthermore, the effects described in the present specification are merely examples and are not limited, and other effects may be exhibited.

5. Effects According to Present Disclosure

As described above, the information processing device (the server device 100 or 100A in the embodiment) according to the present disclosure includes an acquisition unit (the acquisition unit 131 in the embodiment) and an execution unit (the execution unit 134 in the embodiment). The acquisition unit acquires the first speech information indicating the first speech by the user, the second speech information indicating the second speech by the user after the first speech, and the respiration information regarding the respiration of the user. The execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control according to the respiratory state of the user based on the respiration information acquired by acquisition unit.

As described above, the information processing device according to the present disclosure can concatenate the intermittent speeches of the user by executing the processing of concatenating the first speech and the second speech after the first speech by executing the voice interaction control according to the respiratory state of the user. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the information processing device (the server device 100 in the embodiment) according to the present disclosure includes a calculation unit (the calculation unit 132 in the embodiment). The calculation unit calculates the index value indicating the respiratory state of the user using the respiration information. In the case where the index value satisfies the condition, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control. As described above, the information processing device calculates the index value indicating the respiratory state of the user, and in the case where the calculated index value satisfies the condition, the information processing device appropriately enables a plurality of speeches of the user to be concatenated by executing the processing of concatenating the first speech and the second speech by executing the voice interaction control.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control in the case where the comparison result between the index value and the threshold satisfies the condition. As described above, in the case where the comparison result between the index value and the threshold satisfies the condition, the information processing device appropriately enables a plurality of speeches of the user to be concatenated by executing the processing of concatenating the first speech and the second speech by executing the voice interaction control.

Furthermore, the information processing device (the server device 100A in the embodiment) according to the present disclosure includes a calculation unit (the calculation unit 132A in the embodiment). The calculation unit calculates the vector indicating the respiratory state of the user using the respiration information. In the case where the vector satisfies the condition, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control. As described above, the information processing device calculates the vector indicating the respiratory state of the user, and in the case where the calculated vector satisfies the condition, the information processing device appropriately enables a plurality of speeches of the user to be concatenated by executing the processing of concatenating the first speech and the second speech by executing the voice interaction control.

Furthermore, in the case where the vector falls outside the normal range, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control. As described above, in the case where the vector falls outside the normal range, the information processing device appropriately enables a plurality of speeches of the user to be concatenated by executing the processing of concatenating the first speech and the second speech by executing the voice interaction control.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for extending the timeout time regarding voice interaction. As described above, the information processing device can appropriately concatenate speeches even in the case where the user's speech is intermittent for a long time in a case where the user gets out of breath due to exercise or the like by extending the timeout time for the voice interaction. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for extending the timeout time to be used for the voice recognition speech end determination. As described above, because the time for the voice recognition speech end determination is extended, the information processing device can appropriately concatenate speeches even in the case where the user's speech is intermittent for a long time in a case where the user gets out of breath due to exercise or the like by extending the timeout time to be used for the voice recognition speech end determination. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the execution unit executes the processing of concatenating the second speech information indicating the second speech by the user and the first speech before the extended timeout time elapses from the first speech by executing the voice interaction control for extending the timeout time to the extended timeout time. In this manner, the information processing device can concatenate the first speech and the second speech made after the first speech and before the extended timeout time elapses by extending the timeout time related to the voice interaction. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech according to the semantic understanding processing result of the first speech in the case where the semantic understanding processing result of the second speech is uninterpretable. As described above, in the case where the semantic understanding processing result of the second speech is uninterpretable, the information processing device can appropriately concatenate the speeches by concatenating the first speech and the second speech according to the semantic understanding processing result of the first speech. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated, and can increase the possibility of making an uninterpretable speech interpretable.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech with an uninterpretable semantic understanding processing result and the second speech with an uninterpretable semantic understanding processing result. As described above, in the case where the uninterpretable speeches are in succession, the information processing device can appropriately concatenate speeches by concatenating the first speech and the second speech. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated, and can increase the possibility of making an uninterpretable speech interpretable.

Furthermore, the acquisition unit acquires the third speech information indicating the third speech by the user after the second speech. In the case where the semantic understanding processing result of the third speech is uninterpretable, the execution unit executes processing of concatenating the second speech and the third speech. As described above, in the case where the semantic understanding processing result of the third speech is uninterpretable, the information processing device can appropriately concatenate the speeches by executing the processing of concatenating the second speech and the third speech. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated, and can increase the possibility of making an uninterpretable speech interpretable.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech in the case where the first component that is spoken last in the first speech and the second component that is spoken first in the second speech satisfy the condition regarding co-occurrence. As described above, in the case where the last component of the first speech and the first component of the second speech satisfy the condition regarding co-occurrence, the information processing device can appropriately concatenate speeches that are highly likely to be successive in terms of content by concatenating the first speech and the second speech. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech in the case where the probability that the second component appears next to the first component is equal to or larger than a specified value. As described above, in the case where the probability that the second component appears next to the first component is equal to or larger than the specified value, in the case of executing the processing of concatenating the first speech and the second speech by executing the voice interaction control of concatenating the first speech and the second speech, the information processing device can appropriately concatenate speeches that are highly likely to be successive in terms of content by concatenating the first speech and the second speech. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech in the case where the probability that the second component appears next to the first component is equal to or larger than a specified value in the speech history of the user. In this manner, the information processing device can appropriately concatenate the speeches in consideration of the tendency of the user's speech by using the user's speech history. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the acquisition unit acquires the third speech information indicating the third speech by the user after the second speech. In the case where a component spoken last in the second speech and a component spoken first in the third speech satisfy a condition regarding co-occurrence, the execution unit executes the processing of concatenating the second speech and the third speech. As described above, in the case where the last component of the second speech and the first component of the third speech satisfy the condition regarding co-occurrence, the information processing device can appropriately concatenate speeches by executing the processing of concatenating the second speech and the third speech. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

Furthermore, the acquisition unit acquires the respiration information including the displacement amount of the respiration of the user. As described above, by using the displacement amount of the respiration of the user, the Information processing device can more accurately take into account the respiratory state of the user and can concatenate a plurality of speeches of the user.

Furthermore, the acquisition unit acquires the respiration information including the cycle of the respiratory of the user. As described above, by using the cycle of the respiration of the user, the Information processing device can more accurately take into account the respiratory state of the user and can concatenate a plurality of speeches of the user.

Furthermore, the acquisition unit acquires the respiration information including the rate of the respiratory of the user. As described above, by using the rate of the respiration of the user, the information processing device can more accurately take into account the respiratory state of the user and can concatenate a plurality of speeches of the user.

Furthermore, the execution unit does not execute the voice interaction control in the case where the respiratory state of the user is the normal state. As described above, in the case where the respiratory state of the user is the normal state, the information processing device can suppress the influence of the voice interaction control on the processing in the normal state by not executing the voice interaction control and by performing the normal voice recognition processing in the case where the respiration of the user is normal. Therefore, the information processing device appropriately enables a plurality of speeches of the user to be concatenated.

6. Hardware Configuration

The information device such as the server device 100 or 100A or the terminal device 10 according to each of the above-described embodiments is implemented by a computer 1000 having a configuration as illustrated in FIG. 24, for example. FIG. 24 is a hardware configuration diagram illustrating an example of the computer 1000 that implements functions of an information processing device. Hereinafter, the server device 100 according to the first embodiment will be described as an example. The computer 1000 includes a CPU 1100, a RAM 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input/output interface 1600. The units of the computer 1000 are connected by a bus 1050.

The CPU 1100 operates on the basis of programs stored in the ROM 1300 or the HDD 1400, and controls each unit. For example, the CPU 1100 expands the programs stored in the ROM 1300 or the HDD 1400 to the RAM 1200, and executes processing corresponding to various programs.

The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on hardware of the computer 1000, and the like.

The HDD 1400 is a computer-readable recording medium that non-transiently records the programs executed by the CPU 1100, data used by the programs, and the like. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of program data 1450.

The communication interface 1500 is an interface for the computer 1000 to be connected to an external network 1550 (for example, the Internet). For example, the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.

The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. Furthermore, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input/output interface 1600. Furthermore, the input/output interface 1600 may function as a media interface that reads a program or the like recorded in a predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.

For example, in a case where the computer 1000 functions as the server device 100 according to the first embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 and the like by executing the information processing program loaded on the RAM 1200. Furthermore, the HDD 1400 stores the information processing program according to the present disclosure and data in the storage unit 120. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, these programs may be acquired from another device via the external network 1550.

Note that the present technology can also have the following configurations.

(1)

An information processing device including:

an acquisition unit configured to acquire first speech information indicating a first speech by a user, second speech information indicating a second speech by the user after the first speech, and respiration information regarding respiration of the user; and

an execution unit configured to execute processing of concatenating the first speech and the second speech by executing voice interaction control according to a respiratory state of the user based on the respiration information acquired by the acquisition unit.

(2)

The information processing device according to (1), further including:

a calculation unit configured to calculate an index value indicating the respiratory state of the user using the respiration information, in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control in a case where the index value satisfies a condition.

(3)

The information processing device according to (2), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control in a case where a comparison result between the index value and a threshold satisfies the condition.

(4)

The information processing device according to (1), further including:

a calculation unit configured to calculate a vector indicating the respiratory state of the user using the respiration information, in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control in a case where the vector satisfies a condition.

(5)

The information processing device according to (4), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control in a case where the vector is out of a normal range.

(6)

The information processing device according to any one of (1) to (5), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for extending a timeout time regarding voice interaction.

(7)

The information processing device according to (6), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for extending the timeout time to be used for voice recognition speech end determination.

(8)

The information processing device according to (7), in which

the execution unit

executes processing of concatenating the second speech information indicating the second speech and the first speech by the user before an extended timeout time elapses from the first speech by executing the voice interaction control for extending the timeout time to the extended timeout time.

(9)

The information processing device according to any one of (1) to (8), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech according to a semantic understanding processing result of the first speech in a case where a semantic understanding processing result of the second speech is uninterpretable.

(10)

The information processing device according to (9), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech with an uninterpretable semantic understanding processing result and the second speech with an uninterpretable semantic understanding processing result.

(11)

The information processing device according to (9) or (10), in which

the acquisition unit

acquires third speech information indicating a third speech by the user after the second speech, and

the execution unit

executes processing of concatenating the second speech and the third speech in a case where a semantic understanding processing result of the third speech is uninterpretable.

(12)

The information processing device according to any one of (1) to (11), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech in a case where a first component that is spoken last in the first speech and a second component that is spoken first in the second speech satisfy a condition regarding co-occurrence.

(13)

The information processing device according to (12), in which

the execution unit

executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech in a case where a probability that the second component appears next to the first component is equal to or larger than a specified value.

(14)

The information processing device according to (12) or (13), in which

the execution unit executes the processing of concatenating the first speech and the second speech by executing the voice interaction control for concatenating the first speech and the second speech in a case where the probability that the second component appears next to the first component in a speech history of the user is equal to or larger than a specified value.

(15)

The information processing device according to any one of (12) to (14), in which

the acquisition unit

acquires third speech information indicating a third speech by the user after the second speech; and

the execution unit

executes processing of concatenating the second speech and the third speech in a case where a component spoken last in the second speech and a component spoken first in the third speech satisfy a condition regarding co-occurrence.

(16)

The information processing device according to any one of (1) to (15), in which

the acquisition unit

acquires the respiration information including a displacement amount of respiration of the user.

(17)

The information processing device according to any one of (1) to (16), in which

the acquisition unit

acquires the respiration information including a cycle of respiration of the user.

(18)

The information processing device according to any one of (1) to (17), in which

the acquisition unit

acquires the respiration information including a rate of respiration of the user.

(19)

The information processing device according to any one of (1) to (18), in which

the execution unit

does not execute the voice interaction control in a case where the respiratory state of the user is a normal state.

(20)

An information processing method of executing processing including:

acquiring first speech information indicating a first speech by a user, second speech information indicating a second speech by the user after the first speech, and respiration information regarding respiration of the user; and

executing processing of concatenating the first speech and the second speech by executing voice interaction control according to a respiratory state of the user based on the acquired respiration information.

REFERENCE SIGNS LIST

1 Information processing system

100, 100A Server device (information processing device)

110 Communication unit

120, 120A Storage unit

121 Respiration information storage unit

122 User information storage unit

123 Threshold information storage unit

123A Determination information storage unit

124 Functional information storage unit

130, 130A Control unit

131 Acquisition unit

132, 132A Calculation unit

133, 133A Determination unit

134 Execution unit

135 Transmission unit

10 Terminal device

11 Communication unit

12 Input unit

13 Output unit

14 Storage unit

15 Control unit

151 Reception unit

152 Execution unit

153 Acceptance unit

154 Transmission unit

16 Display unit

17 Sensor unit

171 Respiration sensor