Valve Patent | Brain-Computer Interfaces For Computing Systems
Patent: Brain-Computer Interfaces For Computing Systems
Publication Number: 20200298100
Publication Date: 20200924
Applicants: Valve
Abstract
Various embodiments are directed towards employing one or more physical sensors arranged on or in proximity to a video game player to obtain biofeedback measures that are then useable to dynamically modify a state of play of a video game. The sensors may be connected or even unconnected to the game player, replace, or otherwise augment traditional physical game controllers. The sensors gather various biofeedback measures and provide such measures to a biofeedback application programming interface (API). Before and/or during video game play, the video game queries the biofeedback API to request inferences about the game player’s internal state. Responses are then used to modify the state of the video game play. Where the video game is a multi-player video game, biofeedback measures from other game players may also be obtained and used to further modify the state of the video game play.
TECHNICAL FIELD
[0001] The present disclosure relates generally to interactive video games and more particularly, but not exclusively, to brain-computer interfaces for computing systems.
BACKGROUND
[0002] Today, the computer gaming industry is a multi-billion dollar industry. Such popularity may be due in part to faster computing devices, higher quality graphics, and better quality games. Many of today’s video games provide a variety of different input/output devices useable by a game player to interact with the game. For example, many video games allow a player to interact using a keyboard and/or mouse. While such input/output controllers allow the game player to interact with the game, the game player might not feel that they are immersed into the game. Therefore, many video games have been redesigned to allow use of gamepads, joysticks, trackballs, game paddles, and the like to provide ways of immersing the video game player. Some joysticks and/or paddles are configured to resemble a type of device consistent with the video game being played. For example, for some flight simulation games, a joystick might be designed to provide throttle quadrants, levels, wheels, and handheld sticks that appear to the game player as though they are flying within a cockpit of an aircraft.
[0003] By modifying the input devices, video game players are more likely to become involved with and therefore, enjoy the video game more so. As such, the video game player is more likely to continue to play the game, share the game with others, and perhaps to purchase similar games in the future. This trend of modifying the input devices to have more game player involvement is even more apparent with the advent of wireless controllers. For example, in one popular video game, the game input controller is a wireless handheld controller that may include built-in accelerometers, infrared detectors, or similar components. Such components are used to sense a position of the controller in three-dimensional space when pointed at a light emitting diode (LED) within a remote sensor bar. The game player then controls the game using physical gestures as well as traditional buttons, to play games such as bowling, imaginary musical instruments, boxing games, or the like.
[0004] However, while many game players may feel this provides an increased level of involvement in the video game, other game players may still feel that the involvement in the video game is incomplete. Thus, it is with respect to these considerations, and others, that the present disclosure has been made.
BRIEF SUMMARY
[0005] A video game device may be summarized as including: one or more physical biofeedback sensors; at least one nontransitory processor-readable storage medium that stores at least one of data and instructions; and at least one processor operatively coupled to the at least one nontransitory processor-readable storage medium and the one or more physical biofeedback sensors, in operation, the at least one processor: provides game play to a video game player via a user interface that provides functionality for a video game, the game play comprising a plurality of individual components; receives, from the one or more physical biofeedback sensors, biofeedback measures for the video game player while the video game player is playing the video game; processes the biofeedback measures to determine responses of the video game player to the plurality of individual components during the game play of the video game; and modifies or augments the game play of the video game based at least in part on the determined responses of the video game player.
[0006] To process the biofeedback measures, the at least one processor may apply at least one learned model. To process the biofeedback measures, the at least one processor may apply at least one of a Fourier transform or a spectral density analysis. The at least one learned model may have been trained to determine a particular subset of individual components of the plurality individual components that cause the video game player to have a particular cognitive state. The plurality of individual components may include at least one of a game character, a chat message, a weapon, a character selection, an action of a character, an event associated with a character, or a characteristic of another video game player. The one or more physical biofeedback sensors may include one or more electroencephalography (EEG) electrodes, and the biofeedback measures may include EEG signals. The one or more physical biofeedback sensors may include one or more electrodes, and the biofeedback measures may include nerve signals. The biofeedback measures may include at least one of nerve signals, EEG signals, EMG signals, EOG signals, fNIR signals, signals indicative of blood flow, functional near-infrared spectroscopy (fNIR) spectroscopy signals, force-sensitive resistor (FSR) signals, facial expression detection signals, pupil dilation indication signals, eye movement signals, or gestural motion signals. The at least one processor may determine relative weightings of the contributions of the individual components on the determined responses. At least one of the one or more physical biofeedback sensors may be incorporated into a head-mounted display (HMD) device.
[0007] A video game system may be summarized as including: at least one nontransitory processor-readable storage medium that stores at least one of data and instructions; and at least one processor operatively coupled to the at least one nontransitory processor-readable storage medium, in operation, the at least one processor: provides game play to a population of video game players via respective user interfaces that provides functionality for a video game; receives, from physical biofeedback sensors proximate the video game players, biofeedback measures for the video game players while the video game players are playing the video game, the biofeedback measures being captured during the presentation of a plurality of individual components; analyzes the biofeedback measures to determine a subset of the plurality of individual components that contribute to an overall affect or impression of the population of video game players; and modifies or augments the video game responsive to the analysis of the biofeedback measures.
[0008] The plurality of individual components may include at least one of a game character, a chat message, a weapon, a character selection, an action of a character, an event associated with a character, or a characteristic of another video game player. The one or more physical biofeedback sensors may include one or more electroencephalography (EEG) electrodes, and the biofeedback measures may include EEG signals. To analyze the biofeedback measures, the at least one processor may implement at least one model operative to isolate individual components of the plurality of individual components that contribute to the overall affect or impression of the video game players. The at least one processor may receive class information for each of the video game players, and may analyze the biofeedback measures and the class information to determine how different classes of the video game players respond differently to the individual components of the video game. The at least one processor may estimate an opinion of the video game based on the received biofeedback measures. The at least one processor may estimate a lifecycle of the video game based on the received biofeedback measures. The at least one processor may determine a similarity between different portions of the video game based on the received biofeedback measures.
[0009] A video game device may be summarized as including: one or more physical biofeedback sensors; at least one nontransitory processor-readable storage medium that stores at least one of data and instructions; and at least one processor operatively coupled to the at least one nontransitory processor-readable storage medium and the one or more physical biofeedback sensors, in operation, the at least one processor: provides game play to a video game player via a user interface that provides functionality for a video game; receives, from the one or more physical biofeedback sensors, biofeedback measures for the video game player while the video game player is playing the video game; processes the biofeedback measures to determine an internal state of the video game player during the game play of the video game; and modifies or augments the game play of the video game based at least in part on the determined internal state of the video game player.
[0010] The at least one processor may utilize the determined internal state to predict that the video game player is likely to stop playing the video game. The at least one processor may utilize the determined internal state to determine the video game player’s impression of at least one of a weapon, a character, a map, a game mode, a tutorial, a game update, a user interface, a teammate, or a game environment. The biofeedback measures may include at least one of nerve signals, EEG signals, EMG signals, EOG signals, fNIR signals, signals indicative of blood flow, functional near-infrared spectroscopy (fNIR) spectroscopy signals, force-sensitive resistor (FSR) signals, facial expression detection signals, pupil dilation indication signals, eye movement signals, or gestural motion signals. At least one of the one or more physical biofeedback sensors may be incorporated into a head-mounted display (HMD) device.
[0011] The video game device may further include a head-mounted display (HMD) device that carries at least one of the one or more physical biofeedback sensors.
[0012] A video game device may be summarized as including: one or more physical neural stimulators; at least one nontransitory processor-readable storage medium that stores at least one of data and instructions; and at least one processor operatively coupled to the at least one nontransitory processor-readable storage medium and the one or more physical neural stimulators, in operation, the at least one processor: provides game play to a video game player via a user interface that provides functionality for a video game; and provides neural stimulation to the video game player via the one or more physical neural stimulators while the video game player is playing the video game to provide an enhanced experience for the video game player.
[0013] The neural stimulation may provide at least one of: an improvement to the focus of the video game player, an improvement to the memory of the video game player, an improvement to a learning ability of the video game player, a change in the arousal of the video game player, a modification of the vision perception of the video game player, or a modification of the auditory perception of the video game player. The one or more physical neural stimulators may include at least one of a non-invasive neural stimulator or an invasive neural stimulator. The one or more physical neural stimulators may include at least one of a transcranial magnetic stimulation device, a transcranial electrical stimulation device, a microelectrode-based device, or an implantable device. The one or more physical neural stimulators may be operative to provide at least one of sensory stimulation or motor stimulation.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following drawings. In the drawings, like reference numerals refer to like parts throughout the various figures unless otherwise specified.
[0015] For a better understanding of the present disclosure, reference will be made to the following Detailed Description, which is to be read in association with the accompanying drawings wherein:
[0016] FIG. 1 shows a pictorial block diagram illustrating one embodiment of an environment suitable for implementing one or more features of the present disclosure;
[0017] FIG. 2 shows one embodiment of a client device for use in the environment of FIG. 1;
[0018] FIG. 3 shows one embodiment of a network device for use in the environment of FIG. 1;
[0019] FIG. 4 illustrates a flow chart for one embodiment of a process of employing biofeedback measurements from a game player to modify a game play state in a video game;
[0020] FIG. 5 illustrates a flow chart for one embodiment of a process of performing an analysis of biofeedback measures from a game player for use in the video game;
[0021] FIG. 6 illustrates one embodiment of a non-exhaustive, non-limiting example of queries for use in querying a biofeedback application programming interface (API) for biofeedback measures;
[0022] FIG. 7 illustrates one embodiment of a non-exhaustive non-limiting example of using biofeedback measures for use in modifying a game play state in an arena combat video game;
[0023] FIG. 8 illustrates one embodiment of a non-exhaustive non-limiting example of using biofeedback measures for use in modifying a game play state in a space video game;
[0024] FIG. 9 illustrates a flow chart for one embodiment of a process of dynamically modifying or augmenting game play of a video game based on a tracked gaze location of a video game player;
[0025] FIG. 10 illustrates a flow chart for one embodiment of a process of detecting upcoming movements of a user of a user interface;
[0026] FIG. 11 illustrates a flow chart for one embodiment of a process of updating or training a model that is operative to detect upcoming movements of user of a user interface;
[0027] FIG. 12 shows a pictorial block diagram illustrating one embodiment of an environment suitable for implementing one or more features of the present disclosure;* and*
[0028] FIG. 13 illustrates a flow chart for one embodiment of a process of adapting a user interface to remedy difficulties of a user operating a user interface by analyzing biofeedback measures.
[0029] FIG. 14 illustrates a flow chart for one embodiment of a process of performing an analysis of biofeedback measures from a user operating a video game device to determine responses of the user to a plurality of individual components during the game play of the video game.
[0030] FIG. 15 illustrates a flow chart for one embodiment of a process of performing an analysis of biofeedback measures from a population of users operating a video game system to modify or augment a video game.
[0031] FIG. 16 illustrates a flow chart for one embodiment of a process of performing an analysis of biofeedback measures from a user operating a video game system to determine an internal state of the user and to modify or augment a video game.
[0032] FIG. 17 illustrates a flow chart for one embodiment of a process of providing neural stimulation to a user during video game play of a video game system to enhance the user’s gaming experience.
[0033] FIG. 18 is an illustration that shows non-limiting example mechanisms for inducing, writing or otherwise creating signals in a brain of a user (e.g., video game player) to enhance the user’s experience.
[0034] FIG. 19 is an illustration that shows various potential features of a brain-computer interface (BCI) according to embodiments of the present disclosure.
[0035] FIG. 20 is a diagram that shows inputs that cause neuronal firing, including sensory perception, internal cognition, and external influence.
[0036] FIG. 21 is a diagram that shows a BCI with various features of the present disclosure that may be implemented to provide an enhanced experience for a video game player.
DETAILED DESCRIPTION
[0037] One or more implementations of the present disclosure are now described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific exemplary embodiments. The implementations of the present disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. Among other things, one or more implementations may be embodied as methods or devices. Accordingly, embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. The following detailed description is, therefore, not to be taken in a limiting sense.
[0038] Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment, though it may. Furthermore, the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment, although it may. Thus, as described below, various embodiments may be readily combined, without departing from the scope or spirit of the present disclosure.
[0039] In addition, as used herein, the term “or” is an inclusive “or” operator, and is equivalent to the term “and/or,” unless the context clearly dictates otherwise. The term “based on” is not exclusive and allows for being based on additional factors not described, unless the context clearly dictates otherwise. In addition, throughout the specification, the meaning of “a,” “an,” and “the” include plural references. The meaning of “in” includes “in” and “on.”
[0040] As used herein, the terms “biofeedback,” and “physiological” refer to measures of a game player’s specific and quantifiable bodily functions. Such biofeedback measures are typically also referred to as measurements of unconscious or involuntary bodily functions. Such biofeedback measures may include, but are not limited to blood pressure, heart rates, eye movements, pupil dilations, skin temperatures, sweat gland activity, muscle tensions, neuronal activity, other measures discussed herein, etc. As described further herein, such measures are useable to make inferences about the game player’s state of arousal or emotional state. It is noted that a state of arousal includes not only an emotional state, but a physiological state as well. Moreover, as used herein, a state of arousal further includes determination of engagement, valence, and/or other user states based on physiological measurements. As used herein, brain-computer interface (BCI) refers to a communication pathway that translates neuronal signals into actionable input for an external system.
[0041] The following briefly describes the embodiments in order to provide a basic understanding of some aspects of the present disclosure. This brief description is not intended as an extensive overview. It is not intended to identify key or critical elements, or to delineate or otherwise narrow the scope. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
[0042] Briefly stated, various embodiments are directed towards employing one or more physical sensors arranged on or in proximity to a video game player to obtain biofeedback measures about the game player that is useable to dynamically modify a state of play of the video game or to provide other functionality. In one embodiment, the modifications may be performed substantially in real-time. In another embodiment, the modifications may be performed for use in a subsequent game play. The physical sensors may be connected to the game player, and in some implementations may replace and/or otherwise augment traditional physical game controllers. In another embodiment, the physical sensors need not be connected to the game player and may instead be located in proximity to the game player. Non-limiting examples of such physically unconnected sensors include a video camera, an eye tracking system, weight/position sensor pads upon which the game player might stand upon, or the like. The sensors are arranged to gather various biofeedback measures such as heart activity, galvanic skin responses, body temperatures, eye movements, head or other body movements, or the like, and to provide such measures to a biofeedback application programming interface (API). Before and/or during a video game play, the video game may query the biofeedback API for an inference about the game player’s state of arousal, emotional state, cognitive state, or the like, as described further below based on the biofeedback measures. Then, based on the response to the query, the video game modifies a state of video game play. In this manner, the video game may determine whether the game player’s current physiological state is consistent with a type and/or level of experience the video game may seek to provide. For example, if the game player’s stress or arousal state is determined to be above a given threshold, the video game may modify the state of the game play to provide the game player an opportunity to relax and/or recover. In another embodiment, where the game player’s stress or arousal state is determined to be below another threshold, the video game may modify the state of the game play to provide an increased level of excitement for the game player.
[0043] In one embodiment, the threshold may be based on historical biofeedback measures and/or inferences about the particular game player. In another embodiment, the threshold may be based on analysis of the particular game player for the current video game play. In still another embodiment, the threshold may be based on statistical analysis of a plurality of game players.
[0044] In one embodiment, where the video game is configured as a multi-player video game, biofeedback measures from other game players may also be obtained and used to further modify a state of the video game play.
Illustrative Operating Environment
[0045] FIG. 1 illustrates a block diagram generally showing an overview of one embodiment of a system in which one or more features of the present disclosure may be practiced. System 100 may include fewer or more components than those shown in FIG. 1. However, the components shown are sufficient to disclose an illustrative embodiment. As shown in the figure, system 100 includes local area networks (“LANs”)/wide area networks (“WANs”)-(network) 105, wireless network 111, client device 101, game server device (GSD) 110, and biofeedback sensors 120.
[0046] One embodiment of a client device usable as client device 101 is described in more detail below in conjunction with FIG. 2. Briefly, however, client device 101 may include virtually any mobile computing device capable of receiving and sending a message over a network, such as network 111, or the like. Such devices include portable devices such as, radio frequency (RF) devices, infrared (IR) devices, Personal Digital Assistants (PDAs), game consoles, handheld computers, laptop computers, wearable computers, tablet computers, integrated devices combining one or more of the preceding devices, or the like. Client device 101 may also include virtually any computing device that typically connects using a wired communications medium, such as network 105, such as personal computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, or the like. Thus, in one embodiment, client device 101 may be configured to operate over a wired and/or a wireless network.
[0047] Client device 101 typically range widely in terms of capabilities and features. For example, a handheld device may have a numeric keypad and a few lines of monochrome LCD display on which only text may be displayed. In another example, a web-enabled client device may have a touch sensitive screen, a stylus, and several lines of color LCD display in which both text and graphics may be displayed.
[0048] A web-enabled client device may include a browser application that is configured to receive and to send web pages, web-based messages, or the like. The browser application may be configured to receive and display graphics, text, multimedia, or the like, employing virtually any web based language, including a wireless application protocol messages (WAP), or the like. In one embodiment, the browser application is enabled to employ Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SMGL), HyperText Markup Language (HTML), eXtensible Markup Language (XML), or the like, to display and send information.
[0049] Client device 101 also may include at least one application that is configured to receive content from another computing device. The application may include a capability to provide and receive textual content, multimedia information, components to a computer application, such as a video game, or the like. The application may further provide information that identifies itself, including a type, capability, name, or the like. In one embodiment, client device 101 may uniquely identify themselves through any of a variety of mechanisms, including a phone number, Mobile Identification Number (MIN), an electronic serial number (ESN), mobile device identifier, network address, or other identifier. The identifier may be provided in a message, or the like, sent to another computing device.
[0050] Client device 101 may also be configured to communicate a message, such as through email, Short Message Service (SMS), Multimedia Message Service (MMS), instant messaging (IM), internet relay chat (IRC), Mardam-Bey’s IRC (mIRC), Jabber, or the like, between another computing device. However, the present disclosure is not limited to these message protocols, and virtually any other message protocol may be employed. Thus, in one embodiment, client device 101 may enable users to participate in one or more messaging sessions, such as a chat session, a gaming session with messaging, or the like. Such messaging sessions may be text oriented, in that the communications are achieved using text. However, other messaging sessions may occur using client device 101 that employ other mechanisms to communicate, include, but not limited to audio, graphics, video, and/or a combination of text, audio, graphics, and/or video.
[0051] Client device 101 may be configured to receive messages, images, and/or other biofeedback measures, from various biofeedback sensors 120. Illustrated in FIG. 1 are non-limiting, non-exhaustive examples of possible physical biofeedback sensors 120 that may be connected or unconnected to the user, replace, and/or otherwise augment traditional physical game controllers. Thus, as illustrated biofeedback sensors 120 may be integrated within a game controller (sensor 123), one or more keys, wheels, or the like, on a keyboard (sensor 124). In one embodiment, the game controller may include modular and/or pluggable components that may include modular and/or pluggable sensors (123).
[0052] Similarly, biofeedback sensors 120 may include a camera 121, a touch pad 122, or even a head device 125 (e.g., incorporated into a head-mounted display (HMD) device). However, as noted, other biofeedback sensors 120 may also be employed, including, eyeglasses, wrist bands, finger sensor attachments, sensors integrated within or on a computer mice, microphones for measuring various voice patterns, or the like. Thus, it should be apparent to one skilled in the art that various embodiments may employ virtually any mechanism configurable to obtain biofeedback measures of the game player.
[0053] The biofeedback sensors 120 may be arranged to gather various measures of a game player before, after, and/or during a video game play. Such measures include, but are not limited to heart rate and/or heart rate variability; galvanic skin responses; body temperature; eye movement; head, face, hand, or other body movement, gestures, positions, facial expressions, postures, facial strain, or the like. Additionally, biofeedback sensors 120 may collect other measures, including, blood oxygen levels, other forms of skin conductance levels, respiration rate, skin tension, voice stress levels, voice recognition, blood pressure, Electroencephalography (EEG) measures, Electromyography (EMG) measures, response times, Electrooculography (EOG), blood flow (e.g., via an IR camera), functional near-infrared spectroscopy (fNIR) spectroscopy, force-sensitive resistor (FSR), or the like.
[0054] Biofeedback sensors 120 may provide the measures to client device 101. In one embodiment, the measures may be provided to client device 101 over any of a variety of wired and/or wireless connections. Thus, biofeedback measures may be communicated over various cables, wires, or the like, with which other information may also be communicated for a game play. For example, biofeedback measures might be transmitted over a USB cable, coaxial cable, or the like, with which a mouse, keyboard, game controller, or the like, is also coupled to client device 101. However, in another embodiment, a distinct wired connection may be employed. Similarly, biofeedback sensors 120 may employ various wireless connections to communicate biofeedback measures. In addition, any of a variety of communication protocols may be used to communicate the measures. Thus, the present disclosure is not to be construed as being limited to a particular wired or wireless communication mechanism and/or communication protocol.
[0055] In one embodiment, client device 101 may include a biofeedback device interface (BFI) that is configured to determine whether one or more physical sensors 120 are operational, and to manage receipt of biofeedback measures from physical sensors 120. One embodiment of a BFI is described in more detail below in conjunction with FIG. 2. Briefly, however, the BFI may further timestamp the received biofeedback measures, buffer at least some of the measures, and/or forward the measures to GSD 110 for use in modifying a state of a current or future video game play. Buffering of the received biofeedback measures may enable the BFI to perform quality analysis upon the received measures, and to provide alert messages based on a result of the analysis.
[0056] Wireless network 111 is configured to couple client device 101 with network 105. Wireless network 111 may include any of a variety of wireless sub-networks that may further overlay stand-alone ad-hoc networks, or the like, to provide an infrastructure-oriented connection for client device 101. Such sub-networks may include mesh networks, Wireless LAN (WLAN) networks, cellular networks, or the like.
[0057] Wireless network 111 may further include an autonomous system of terminals, gateways, routers, or the like connected by wireless radio links, or the like. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, such that the topology of wireless network 111 may change rapidly.
[0058] Wireless network 111 may further employ a plurality of access technologies including 2nd (2G), 3rd (3G), 4th (4G) generation radio access for cellular systems, WLAN, Wireless Router (WR) mesh, or the like. Access technologies such as 2G, 2.5G, 3G, 4G, and future access networks may enable wide area coverage for client devices, such as client device 101 with various degrees of mobility. For example, wireless network 111 may enable a radio connection through a radio network access such as Global System for Mobile communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Bluetooth, or the like. In essence, wireless network 111 may include virtually any wireless communication mechanism by which information may travel between client device 101 and another computing device, network, or the like.
[0059] Network 105 is configured to couple computing devices, such as GSD 110 to other computing devices, including potentially through wireless network 111 to client device 101. However, as illustrated, client device 101 may also be connected through network 105 to GSD 110. In any event, network 105 is enabled to employ any form of computer readable media for communicating information from one electronic device to another. Also, network 105 can include the Internet in addition to local area networks (LANs), wide area networks (WANs), direct connections, such as through a universal serial bus (USB) port, other forms of computer-readable media, or any combination thereof. On an interconnected set of LANs, including those based on differing architectures and protocols, a router acts as a link between LANs, enabling messages to be sent from one to another. Also, communication links within LANs typically include twisted wire pair or coaxial cable, while communication links between networks may utilize analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines (DSLs), wireless links including satellite links, or other communications links known to those skilled in the art. Furthermore, remote computers and other related electronic devices could be remotely connected to either LANs or WANs via a modem and temporary telephone link. In essence, network 105 includes any communication method by which information may travel between computing devices.
[0060] One embodiment of GSD 110 is described in more detail below in conjunction with FIG. 3. Briefly, however, GSD 110 may include any computing device capable of connecting to network 105 to enable a user to participate in one or more online games, including, but not limited multi-player games, as well as single player games. Thus, while FIG. 1 illustrates a single client device 101 with biofeedback sensors 120, the present disclosure is not so limited, and a plurality of similar client devices with biofeedback sensors may be deployed within system 100.
[0061] Therefore, GSD 110 is configured to receive various biofeedback measures from one or more game players and to employ the received measures to modify a state of the video game. GSD 110 may employ the biofeedback to dynamically adjust a game play difficulty, and/or other aspects of a video game based on the biofeedback measures. For example, in one embodiment, if it is determined that a user is experiencing a level of stress defined as excessive, based on a threshold, the video game within GSD 110 might provide a different game play to enable reduction in the determined stress level.
[0062] GSD 110 may also enable the video game to provide a unique experience each time it is played based on the biofeedback measures of the game player. For example, in one embodiment, a color of an object, size, shape, and/or action of a game character, or the like, may be adjusted based on biofeedback measures. That is various aspects of a background displayed within the background of the game may be modified based on the results of an analysis of the biofeedback measures.
[0063] In one embodiment, historical measurements may be stored, and analyzed to enable GSD 110 to detect a particular game player or to modify current game play for a particular game player. Such stored measurements may then be used to personalize the game play for the particular game player, identify changes in a game play by the particular game player based on a determined trend determination, or the like. In one embodiment, historical measurements together with analysis of the biofeedback measures may be used to determine whether the game player is currently associated with a prior user profile–that is, whether this game player is someone that has played before. GSD 110 may also adjust a type of game play offered based a determination of the game player’s level of engagement during a game play, historical patterns, or the like.
[0064] GSD 110 may further provide matchmaking decisions based in whole or in part on a physiological or emotional state of a game player that may seek a multiplayer game session. In still other embodiments, GSD 110 may dynamically adjust game play instructions, tutorials, or the like, based on the received biofeedback measures. For example, where it might be determined that the game player is determined to be bored or otherwise uninterested in the instructions, tutorials, or the like, GSD 110 might enable the material to be sped up, skipped or the like. Alternatively, where it might be determined based on the biofeedback measures that the game player is confused or is having a difficult time making a decision, tutorials or other guidance may be provided to assist the game player.
[0065] GSD 110 is not limited to these examples of how biofeedback measures may be used however, and others ways of employing the biofeedback measures to modify a game play state may also be used. For example, the biofeedback measures may be employed to directly control an aspect of the game play. One non-limiting example of such is described in more detail below in conjunction with FIG. 8.
[0066] In still other embodiments, GSD 110 may depict the game player’s emotional, physiological state and/or other aspects of the game player’s expression within a game character. For example, a game player’s avatar might be modified to display a heart that beats at the rate of the game player’s heart, or the avatar might be shown to breathe at the game player’s rate, or sweat, or even show a facial expression, or body position based on the received biofeedback measures for the game player. Thus, GSD 110 may employ biofeedback measures in any of a variety of ways to modify a state of a game play.
[0067] Devices that may operate as GSD 110 include personal computers, desktop computers, multiprocessor systems, video game consoles, microprocessor-based or programmable consumer electronics, network PCs, server devices, and the like.
[0068] Moreover, although GSD 110 is illustrated as a single network device the present disclosure is not so limited. For example, one or more of the functions associated with GSD 110 may be implemented in a plurality of different network devices, distributed across a peer-to-peer system structure, or the like, without departing from the scope or spirit of the present disclosure. Thus, as described below in conjunction with FIG. 3, is a network device 300 configured to manage a game play using biofeedback measures to modify a state of the game. However, other configurations are also envisaged.
[0069] For example, in another embodiment, the client device 101 may be configured to include components from GSD 110 such that client device 101 may operate independent of GSD 110. That is, in one embodiment, client device 101 may include game software with biofeedback, biofeedback Application Programming Interfaces (APIs), and the like, and operate without use of a network connection to GSD 110. Client device 101 may therefore, operate as essentially a standalone game device with interfaces to the biofeedback sensors, and other input/output devices for user enjoyment. Therefore, the present disclosure is not constrained or otherwise limited by the configurations shown in the figures.
[0070] Although a single client device 101 is illustrated in FIG. 1 having a single game player and a single set of biofeedback sensors 120, other embodiments are also envisaged. For example, in one embodiment, a plurality of game players, each having their own biofeedback sensors might interact and play together a same video game through the same client device 101 or through multiple client devices connected together via a network. Thus, multi-player configurations may include such variations as multiple game players employing the same or different client devices. Therefore, FIG. 1 is not to be construed as being limited to a single game player configuration.
Illustrative Client Device
[0071] FIG. 2 shows one embodiment of client device 200 that may be included in a system implementing the present disclosure. Client device 200 may include many more or less components than those shown in FIG. 2. For example, client device 200 may be configured with a reduced set of components for use as a standalone video game device. However, the components shown are sufficient to disclose an illustrative embodiment. Client device 200 may represent, for example, client device 101 of FIG. 1.
[0072] As shown in FIG. 2, client device 200 includes a processing unit (CPU) 222 in communication with a mass memory 230 via a bus 224. Client device 200 also includes a power supply 226, one or more network interfaces 250, an audio interface 252 that may be configured to receive an audio input as well as to provide an audio output, a display 254, a keypad 256, an illuminator 258, an input/output interface 260, a haptic interface 262, and a global positioning systems (GPS) receiver 264. Power supply 226 provides power to client device 200. A rechargeable or non-rechargeable battery may be used to provide power. The power may also be provided by an external power source, such as an AC adapter or a powered docking cradle that supplements and/or recharges a battery. Client device 200 may also include a graphical interface 266 that may be configured to receive a graphical input, such as through a camera, scanner, or the like.
[0073] Network interface 250 includes circuitry for coupling client device 200 to one or more networks, and is constructed for use with one or more communication protocols and technologies including, but not limited to, global system for mobile communication (GSM), code division multiple access (CDMA), time division multiple access (TDMA), user datagram protocol (UDP), transmission control protocol/Internet protocol (TCP/IP), SMS, general packet radio service (GPRS), WAP, ultra wide band (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMax), SIP/RTP, Bluetooth, Wi-Fi, Zigbee, UMTS, HSDPA, WCDMA, WEDGE, or any of a variety of other wired and/or wireless communication protocols. Network interface 250 is sometimes known as a transceiver, transceiving device, or network interface card (NIC).
[0074] Audio interface 252 is arranged to produce and receive audio signals such as the sound of a human voice. For example, audio interface 252 may be coupled to a speaker and microphone (not shown) to enable telecommunication with others and/or generate an audio acknowledgement for some action. Display 254 may be a liquid crystal display (LCD), gas plasma, light emitting diode (LED), or any other type of display used with a computing device. Display 254 may also include a touch sensitive screen arranged to receive input from an object such as a stylus or a digit from a human hand.
[0075] Keypad 256 may comprise any input device arranged to receive input from a user. For example, keypad 256 may include a push button numeric dial, or a keyboard. Keypad 256 may also include command buttons that are associated with selecting and sending images, game play, messaging sessions, or the like. In one embodiment, keypad 256 may include various biofeedback sensors arranged to obtain various measures including, but not limited to pressure readings, response time readings, sweat readings, or the like.
[0076] Illuminator 258 may provide a status indication and/or provide light. Illuminator 258 may remain active for specific periods of time or in response to events. For example, when illuminator 258 is active, it may backlight the buttons on keypad 256 and stay on while the client device is powered. Also, illuminator 258 may backlight these buttons in various patterns when particular actions are performed, such as dialing another client device. Illuminator 258 may also cause light sources positioned within a transparent or translucent case of the client device to illuminate in response to actions.
[0077] Client device 200 also comprises input/output interface 260 for communicating with external devices, such as a headset, or other input or output devices, including, but not limited, to joystick, mouse, or the like. As described above in conjunction with FIG. 1, client device 200 may also be configured to communicate with one or more biofeedback sensors through input/output interface 260. Input/output interface 260 can utilize one or more communication technologies, such as USB, infrared, Bluetooth.RTM., or the like. Haptic interface 262 is arranged to provide tactile feedback to a user of the client device. For example, the haptic interface may be employed to vibrate client device 200 in a particular way when another user of a computing device is calling.
[0078] GPS transceiver 264 can determine the physical coordinates of client device 200 on the surface of the Earth, which typically outputs a location as latitude and longitude values. GPS transceiver 264 can also employ other geo-positioning mechanisms, including, but not limited to, triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS or the like, to further determine the physical location of client device 200 on the surface of the Earth. It is understood that under different conditions, GPS transceiver 264 can determine a physical location within millimeters for client device 200; and in other cases, the determined physical location may be less precise, such as within a meter or significantly greater distances. In one embodiment, however, client device 200 may, through other components, provide other information that may be employed to determine a geo physical location of the device, including for example, a MAC address, IP address, or other network address.
[0079] Mass memory 230 includes a RAM 232, a ROM 234, and/or other storage. Mass memory 230 illustrates another example of computer storage media for storage of information such as computer readable instructions, data structures, program modules or other data. Mass memory 230 stores a basic input/output system (“BIOS”) 240 for controlling low-level operation of client device 200. The mass memory also stores an operating system 241 for controlling the operation of client device 200. It will be appreciated that this component may include a general purpose operating system such as a version of UNIX, or LINUX.TM., or a specialized client communication operating system such as Windows Mobile.TM., the Symbian.RTM. operating system, or even any of a variety of video game console operating systems. The operating system may include, or interface with a Java virtual machine module that enables control of hardware components and/or operating system operations via Java application programs.
[0080] Memory 230 further includes one or more data storage 244, which can be utilized by client device 200 to store, among other things, applications and/or other data. For example, data storage 244 may also be employed to store information that describes various capabilities of client device 200, a device identifier, and the like. The capability information may further be provided to another device based on any of a variety of events, including being sent as part of a header during a communication, sent upon request, or the like. Data storage 244 may also be employed to buffer one or more measures received from a biofeedback sensor.
[0081] In one embodiment, data storage 244 may also include cookies, portions of a computer application, user preferences, game play data, messaging data, and/or other digital content, and the like. At least a portion of the stored data may also be stored on an optional hard disk drive 272, optional portable storage medium 270, or other storage medium (not shown) within client device 200.
[0082] Applications 242 may include computer executable instructions which, when executed by client device 200, transmit, receive, and/or otherwise process messages (e.g., SMS, MMS, IMS, IM, email, and/or other messages), audio, video, and enable telecommunication with another user of another client device. Other examples of application programs include calendars, browsers, email clients, IM applications, VOIP applications, contact managers, task managers, database programs, word processing programs, security applications, spreadsheet programs, search programs, and so forth. Applications 242 may further include browser 245, messenger 243, game client 248, and biofeedback device interface (BFI) 249.
[0083] Messenger 243 may be configured to initiate and manage a messaging session using any of a variety of messaging communications including, but not limited to email, Short Message Service (SMS), Instant Message (IM), Multimedia Message Service (MMS), Internet relay chat (IRC), mIRC, VOIP, or the like. For example, in one embodiment, messenger 243 may be configured as an IM application, such as AOL Instant Messenger, Yahoo! Messenger, .NET Messenger Server, ICQ, or the like. In one embodiment, messenger 243 may be configured to include a mail user agent (MUA) such as Elm, Pine, MH, Outlook, Eudora, Mac Mail, Mozilla Thunderbird, or the like. In another embodiment, messenger 243 may be a client application that is configured to integrate and employ a variety of messaging protocols. Moreover, messenger 243 might be configured to manage a plurality of messaging sessions concurrently, enabling a user to communicate with a plurality of different other users in different messaging sessions, and/or a same messaging session. As used herein, the term “active messaging session” refers to a messaging session in which a user may communicate with another user independent of having to restart and/or re-establish the messaging session. Thus, maintaining a messaging session as active indicates that the messaging session is established, and has not been terminated, or otherwise, placed into a sleep mode, or other inactive mode, whereby messages may not be actively sent and/or received.
[0084] Browser 245 may include virtually any client application configured to receive and display graphics, text, multimedia, and the like, employing virtually any web based language. In one embodiment, the browser application is enabled to employ Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SMGL), HyperText Markup Language (HTML), eXtensible Markup Language (XML), and the like, to display and send a message. However, any of a variety of other web based languages may also be employed.
[0085] Game client 248 represents a game application component that is configured to enable a user to select one or more games to play, register for access to the one or more games, and/or launch the one or more games for online interactive play. In one embodiment, game client 248 may establish communications over a network with a network device, such as GSD 110, or the like, to enable registration, purchase, access to, and/or play of the one or more computer games.
[0086] Game client 248 may receive from a user via various user input devices, including, but not limited to those mentioned above, directions to launch a computer game. Game client 248 may then enable communications of game data between client device 200 and the GSD 110, another client device, or the like.
[0087] In one embodiment, game client 248 represents a computer game application; however, game client 248 is not limited to game applications, and may also represent virtually any interactive computer application, or other interactive digital content. Thus, while described herein as employing biofeedback measures to modify a state of a video game play, the present disclosure is not to be construed as being limited to video game play, and states of other applications may also be modified. For example, a presentation, tutorial, or the like, may be modified based on biofeedback measures.
[0088] Thus, in one embodiment, game engine 248 represents a client component useable to enable online multi-user game play, and/or single game player use. Non-exhaustive, non-limiting examples of such computer games include but are not limited to Half-Life, Team Fortress, Portal, Counter-Strike, Left 4 Dead, and Day of Defeat developed by Valve Corporation of Bellevue, Wash.
[0089] BFI 249 is configured to detect a connection of one or more biofeedback sensors, and to collect measures received from such sensors. In one embodiment, BFI 249 may provide information to a remote network device, and/or to game client 248 indicating that a connection with a biofeedback sensor is detected. BFI 249 may further buffer at least some of the received measures. In another embodiment, BFI 249 may select to instead, provide the received measures to the remote network device, absent buffering, virtually in real-time. In one embodiment, BFI 249 may convert the measures into a format and/or protocol usable to communicate the measures over a network to the remote network device. In another embodiment, BFI 249 may select to not communicate the measures over a network, such as when client device 200 may be configured as a standalone type of video game console. In one embodiment, BFI 249 may also time stamp the received measures such that the measures may be readily correlated. Further, BFI 249 may provide a sensor source identifier to the measures so that measures may be distinguished based on its sensor source.
[0090] BFI 249 may further perform one or more analysis on the received measures to determine if a sensor is providing faulty readings, has become disconnected, or the like. Such determinations may be based on a comparison over time of a plurality of received measures for a given sensor to detect changes from an anticipated range of values for a received measure. For example, if BFI 249 detects that the sensor measure is a heart rate sensor, and the measures indicate a heart rate of, for example, 2 beats per minute, or even 100 beats per second, then BFI 249 may determine that the sensor measures are faulty. It should be clear; however, that BFI 249 may employ other range values, and is not constrained to these example range values. Moreover, BFI 249 may employ different range values for different sensors. In one embodiment, BFI 249 might provide the determined faulty measures over the network at least for a given period of time, under an assumption that the game player is temporarily adjusting the sensor. However, in another embodiment, if the sensor is determined to be faulty beyond the given time period, BFI 249 may select to cease transmission of the measures, and/or send a message to the remote network device.
[0091] As noted above, in conjunction with FIG. 1, client device 200 may be configured to include components of network device 300 (described below in conjunction with FIG. 3), including biofeedback APIs, game server components, and the like. In such an embodiment, client device 200 might operate essentially as a standalone game console, without communicating with network device 300. In such a configuration, client device 200 may be termed a standalone video game device.
Illustrative Network Device
[0092] FIG. 3 shows one embodiment of a network device, according to one embodiment. Network device 300 may include many more or fewer components than those shown. The components shown, however, are sufficient to disclose an illustrative embodiment. Network device 300 may represent, for example, GSD 110 of FIG. 1.
[0093] Network device 300 includes processing unit 312, video display adapter 314, and a mass memory, all in communication with each other via bus 322. The mass memory generally includes RAM 316, ROM 332, and one or more permanent mass storage devices, such as hard disk drive 328, and removable storage device 326 that may represent a tape drive, optical drive, and/or floppy disk drive. The mass memory stores operating system 320 for controlling the operation of network device 300. Any general-purpose operating system may be employed. Basic input/output system (“BIOS”) 318 is also provided for controlling the low-level operation of network device 300. As illustrated in FIG. 3, network device 300 also can communicate with the Internet, or some other communications network, via network interface unit 310, which is constructed for use with various communication protocols including the TCP/IP protocol, Wi-Fi, Zigbee, WCDMA, HSDPA, Bluetooth, WEDGE, EDGE, UMTS, or the like. Network interface unit 310 is sometimes known as a transceiver, transceiving device, or network interface card (NIC).
[0094] The mass memory as described above illustrates another type of computer-readable media, namely computer storage media. Computer-readable storage media may include volatile, nonvolatile, removable, and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.
[0095] The mass memory also stores program code and data. In one embodiment, the mass memory may include data store 356. Data stores 356 includes virtually any component that is configured and arranged to store data including, but not limited to game player preferences, game play state and/or other game play data, messaging data, biofeedback measures, and the like. Data store 356 also includes virtually any component that is configured and arranged to store and manage digital content, such as computer applications, video games, and the like. As such, data stores 356 may be implemented using a data base, a file, directory, or the like. At least a portion of the stored data may also be stored on hard disk drive 328, a portable device such as cd-rom/dvd-rom drive 326, or even on other storage mediums (not shown) within network device 300 or remotely on yet another network device.
[0096] One or more applications 350 are loaded into mass memory and run on operating system 320. Examples of application programs may include transcoders, schedulers, calendars, database programs, word processing programs, HTTP programs, customizable user interface programs, IPSec applications, computer games, encryption programs, security programs, VPN programs, SMS message servers, IM message servers, email servers, account management and so forth. Applications 350 may also include web services 346, message server 354, game server with biofeedback (GSB) 352, and Biofeedback APIs (BAPI) 353.
[0097] Web services 346 represent any of a variety of services that are configured to provide content over a network to another computing device. Thus, web services 346 include for example, a web server, messaging server, a File Transfer Protocol (FTP) server, a database server, a content server, or the like. Web services 346 may provide the content over the network using any of a variety of formats, including, but not limited to WAP, HDML, WML, SMGL, HTML, XML, cHTML, xHTML, or the like.
[0098] Message server 354 may include virtually any computing component or components configured and arranged to manage messages from message user agents, and/or other message servers, or to deliver messages to a message application, one another network device. Message server 354 is not limited to a particular type of messaging. Thus, message server 354 may provide capability for such messaging services, including, but not limited to email, SMS, MMS, IM, IRC, mIRC, Jabber, VOIP, and/or a combination of one or more messaging services.
[0099] GSB 352 is configured to manage delivery and play of a video game using biofeedback information obtained from one or more client devices, such as client device 101 of FIG. 1. Typically, GSB 352 may provide components to an application, such as a game application to the client device over a network. In one embodiment, at least one of the components provided is encrypted using any of a variety of encryption mechanisms. For example, in one embodiment, Crypto++, an open-source class library of cryptographic techniques, is employed in encrypting or decrypting components of the application. However, virtually any other encryption and decryption mechanism may be used.
[0100] GSB 352 may further receive and/or authenticate a request from a client device for access to an application. GSB 352 may provide for purchase of an application, such as a computer game, enable registration for play of the application, and/or enable download access for the application.
[0101] GSB 352 may further enable communications between client devices participating in a multi-player application by receiving and/or providing various data, messages, or the like, between the client devices.
[0102] GSB 352 may query Biofeedback APIs (BAPI) 353 for information about one or more game player’s state or arousal, and/or other information about the game player(s). GSB 352 may then modify a state of the video game play based on the received responses to the query. Non-limiting, non-exhaustive examples of queries that GSB 352 might submit to BAPI 353 are described below in conjunction with FIG. 6. Non-limiting, non-exhaustive examples of possible ways in which a video game play might be modified are described below in conjunction with FIGS. 7-8. In one embodiment, GSB 352 may generally employ processes such as described below in conjunction with FIGS. 5-6 to perform at least some of its actions.
[0103] BAPI 353 is configured to perform various analysis from the received biofeedback measures and to provide responses to various queries from GSB 352. In one embodiment, BAPI 353 may collect and store received biofeedback measures in data store 356 to enable data analysis to be performed, auditing over a time period to be performed, historical data to be collected and analyzed, or the like. In one embodiment, BAPI 353 may perform at least some analysis upon the received biofeedback measures substantially in real-time. That is, as soon as the measures are received by BAPI 353, at least some analysis is performed on the measures.
[0104] As noted, BAPI 353 may receive biofeedback measures from a variety of different biofeedback sensors, including, but not limited to those described above in conjunction with FIG. 1. In one embodiment, the received measures may be identified as a sensor source, such as a heart rate sensor, a galvanic skin sensor, or the like.
[0105] BAPI 353, as stated, may perform analysis on the received measures. For example, BAPI 353 may receive raw biofeedback measures, and determine from the measures a heartbeat based on the measures. In another embodiment, BAPI 353 may employ one or more measures to determine other physiological information about an associated game player. For example, BAPI 353 might compute a heart rate variability from heart sensor measures. Similarly, BAPI 353 might compute a standard deviation of heart rate activity over a defined time period, determine a trend over time in a heart rate, and/or determine other heart patterns. BAPI 353 may analyze frequency spectrums of heart rate data, including breaking down beat-to-beat intervals into various frequencies using, for example, Fourier transforms, or similar analysis techniques. BAPI 353 may also employ various measures to determine other physiological information about the game player including, but not limited to respiration rate, relaxation level, fight or flight data, or the like.
[0106] BAPI 353 might store the results of the analysis for use during a subsequent game play, or determine and employ the results, virtually in real-time. BAPI 353 may further perform various recalibration activities, including, such as a progressive recalibration activity. In one embodiment, the recalibration activities may be performed on the sensors, and/or to account for physiological changes over time.
[0107] Similarly, BAPI 353 may employ historical data based on the biofeedback measures to recognize a particular game player, profiles, or the like, through various mechanisms, including, pattern matching, or the like. BAPI 353 may further recognize when one game player disconnects from the sensors and/or is replaced by another game player, based on such activities as missing and/or corrupt biofeedback measures, pattern changes, or the like.
[0108] BAPI 353 may also be configured to detect particular patterns, conditions, or the like from analyzing the received biofeedback measures. For example, in one embodiment, BAPI 353 might detect and/or even predict an onset of motion sickness based, for example, on a causal coherence between a heart rate, blood pressure, and/or other measures. However, BAPI 353 may further detect other situations that may be of a severity that warrants sending of an alert message to the video game player, and/or to GSB 352 to cease game play. However, BAPI 353 is not constrained to these actions, and others may also be performed.
[0109] As noted above, BAPI 353 is further configured to make inferences about a state of arousal, emotional states, or the like, of a game player based on analysis of the received biofeedback measures. Such inferences may be performed based on the measures as received, and/or based on historical data about the game player, and/or other game players. GSB 352 may query BAPI 353 for information about one or more game player’s state or arousal, and/or other information about the game player(s) based in part on the inferences.
[0110] In one embodiment, GSB 352 may send a query request for information about the game player’s state of arousal. In response, BAPI 353 may provide a qualitative response, such as “is happy,” “is sad,” “is stressed,” “is lying,” “is bored,” “is excited,” or the like. However, in another embodiment, the response may be a quantitative response indicating a level of happiness, such as from zero to ten, or the like. However, the present disclosure is not restricted to these values or even to this example range, and clearly, other values and/or ranges may be used. For example, a quantitative response indicating a level of happiness could also be a letter grade.
……
……
……
