Sony Patent | Communication apparatus, generated data size control method, communication method, and program

Patent: Communication apparatus, generated data size control method, communication method, and program

Drawings: Click to check drawins

Publication Number: 20210058601

Publication Date: 20210225

Applicant: Sony

Assignee: Sony Interactive Entertainment Inc.

Abstract

A video data transmission unit transmits data generated by a video data generation unit to an HMD. A sensing data reception unit receives data transmitted from the HMD, the data including size data indicating a size of the data successfully received by the HMD. A required time specification unit specifies time it took for a round-trip communication in half-duplex wireless communication between a relay apparatus and the HMD. A generation control unit controls a size of the data generated by the video data generation unit per unit time based on the specified time, the size indicated in the size data, and a given target time regarding the round-trip communication.

Claims

1. A communication apparatus that performs half-duplex wireless communication with a communication partner apparatus, the communication apparatus comprising: a transmission unit that transmits data generated by a data generation unit to the communication partner apparatus; a reception unit that receives data transmitted from the communication partner apparatus, the data including size data indicating a size of the data successfully received by the communication partner apparatus; a time specification unit that specifies time it took for a round-trip communication in the half-duplex wireless communication with the communication partner apparatus; and a generation control unit that controls a size of the data generated by the data generation unit per unit time based on the specified time, the size indicated in the size data, and a given target time regarding the round-trip communication.

2. The communication apparatus according to claim 1, wherein the time specification unit specifies the time it took for the round-trip communication corresponding to time from the start of the transmission of the data to the communication partner apparatus in a first period to the end of the reception of the data transmitted by the communication partner apparatus in a second period following the first period, and the reception unit receives the size data indicating the size of the data successfully received by the communication partner apparatus in the first period.

3. The communication apparatus according to claim 1, wherein the reception unit receives sensing data transmitted from the communication partner apparatus, the data generation unit generates data of a video, and the generation control unit controls quality of the data of the video.

4. A communication apparatus that performs half-duplex wireless communication with a communication partner apparatus, the communication apparatus comprising: a reception unit that receives data from the communication partner apparatus; and a transmission unit that transmits, to the communication partner apparatus, sensing data including, in a header, size data indicating a size of successfully received data, wherein the transmission unit transmits the sensing data even in a case where the reception of part of the data transmitted by the communication partner apparatus has failed.

5. The communication apparatus according to claim 4, wherein the reception unit receives encoded video data from the communication partner apparatus, the communication apparatus further includes an output unit that outputs the video data to a decode unit, and for part of the video data failed to receive, the output unit outputs the video data supplemented with predetermined data to the decode unit.

6. The communication apparatus according to claim 4, wherein the reception unit receives a plurality of packets provided with sequence numbers in one communication, and when the transmission unit successfully receives a packet with the last sequence number among the plurality of packets, the transmission unit transmits the size data in response to the reception of the plurality of packets even in a case where the reception of at least one of the plurality of packets has failed.

7. A generated data size control method comprising: transmitting data generated by a data generation unit to a communication partner apparatus; receiving data transmitted from the communication partner apparatus, the data including size data indicating a size of the data successfully received by the communication partner apparatus; specifying time it took for a round-trip communication in half-duplex wireless communication with the communication partner apparatus; and controlling a size of the data generated by the data generation unit per unit time based on the specified time, the size indicated in the size data, and a given target time regarding the round-trip communication.

8. A communication method comprising: receiving data from a communication partner apparatus; and transmitting, to the communication partner apparatus, sensing data including, in a header, size data indicating a size of successfully received data, wherein in the transmitting the sensing data, the sensing data is transmitted even in a case where the reception of part of the data transmitted by the communication partner apparatus has failed.

9. A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to carry out actions, comprising: transmitting data generated by a data generation unit to a communication partner apparatus; receiving data transmitted from the communication partner apparatus, the data including size data indicating a size of the data successfully received by the communication partner apparatus; specifying time it took for a round-trip communication in half-duplex wireless communication with the communication partner apparatus; and controlling a size of the data generated by the data generation unit per unit time based on the specified time, the size indicated in the size data, and a given target time regarding the round-trip communication.

10. A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to carry out actions, comprising: receiving data from a communication partner apparatus; and transmitting, to the communication partner apparatus, sensing data including, in a header, size data indicating a size of successfully received data, wherein in the transmitting the sensing data, the sensing data is transmitted even in a case where the reception of part of the data transmitted by the communication partner apparatus has failed.

Description

TECHNICAL FIELD

[0001] The present invention relates to a communication apparatus, a generated data size control method, a communication method, and a program.

BACKGROUND ART

[0002] In recent years, a head-mounted display (HMD) that can be mounted on the head to watch a video displayed in front of the eyes is increasingly used. There is an HMD connected by wire to an entertainment apparatus that executes a program, such as a game program, and the HMD receives and displays a video generated by the entertainment apparatus.

[0003] In addition, there is an HMD that changes the video to be displayed according to the movement of the head of the user. The HMD includes, for example, a motion sensor, a camera, and the like, and the HMD generates sensing data, such as data indicating detection results of the motion sensor and images captured by the camera, and transmits the generated sensing data to the entertainment apparatus. The entertainment apparatus that has received the sensing data generates a video based on the sensing data and transmits the video to the HMD. Furthermore, the HMD displays the received video, and the user can watch the video that changes according to the movement of the head.

SUMMARY

Technical Problems

[0004] The inventors are examining to include, for example, a communication path based on half-duplex wireless communication compliant with a standard, such as wireless fidelity (Wi-Fi) (registered trademark), in a communication path between the entertainment apparatus and the HMD. In the half-duplex wireless communication, transmission of data from an apparatus on the transmission side (hereinafter, referred to as communication apparatus) to an apparatus on the reception side (hereinafter, referred to as communication partner apparatus) and transmission of data from the communication partner apparatus to the communication apparatus are alternately performed. In the example described above, the transmission of the data of the video from the entertainment apparatus to the HMD and the transmission of the sensing data from the HMD to the entertainment apparatus are alternately performed.

[0005] It is desirable that the quality of the data used in the communication partner apparatus, such as the data of the video generated by the entertainment apparatus, be as high as possible. For example, it is desirable that the image quality of the displayed video be as high as possible in the HMD used as described above. Therefore, it is desirable that the size of the data, such as the data of the video generated by the entertainment apparatus, generated per unit be as large as possible.

[0006] On the other hand, it is important to bring the time it takes for a round-trip communication closer to a given target time even by reducing the size of the data generated per unit time in order to reduce the discomfort of the user in the communication that requires promptness as described above.

[0007] Here, the time it takes for a round-trip communication can be brought closer to the given target time by controlling the size of the data generated per unit time based on the size of the data communicated in a round-trip communication and the time it took for the communication.

[0008] However, in a case where, for example, the communication environment is poor, part of the data may be missing, and the missing part may not reach the communication partner. In such a case, the time it takes for a round-trip communication may not be close to the given target time unless the size of the data successfully communicated by the communication partner is taken into account to control the size of the data generated per unit time.

[0009] The present invention has been made in view of the problems, and an object of the present invention is to provide a communication apparatus, a generated data size control method, a communication method, and a program that can bring the time it takes for a round-trip communication in half-duplex wireless communication closer to a given target time.

Solution to Problems

[0010] To solve the problems, the present invention provides a communication apparatus that performs half-duplex wireless communication with a communication partner apparatus, the communication apparatus including a transmission unit that transmits data generated by a data generation unit to the communication partner apparatus, a reception unit that receives data transmitted from the communication partner apparatus, the data including size data indicating a size of the data successfully received by the communication partner apparatus, a time specification unit that specifies time it took for a round-trip communication in the half-duplex wireless communication with the communication partner apparatus, and a generation control unit that controls a size of the data generated by the data generation unit per unit time based on the specified time, the size indicated in the size data, and a given target time regarding the round-trip communication.

[0011] In a mode of the present invention, the time specification unit specifies the time it took for the round-trip communication corresponding to time from the start of the transmission of the data to the communication partner apparatus in a first period to the end of the reception of the data transmitted by the communication partner apparatus in a second period following the first period, and the reception unit receives the size data indicating the size of the data successfully received by the communication partner apparatus in the first period.

[0012] Furthermore, in a mode of the present invention, the reception unit receives sensing data transmitted from the communication partner apparatus, the data generation unit generates data of a video, and the generation control unit controls quality of the data of the video.

[0013] Furthermore, the present invention provides another communication apparatus that performs half-duplex wireless communication with a communication partner apparatus, the communication apparatus including a reception unit that receives data from the communication partner apparatus, and a transmission unit that transmits size data indicating a size of successfully received data to the communication partner apparatus, in which the transmission unit transmits the size data even in a case where the reception of part of the data transmitted by the communication partner apparatus has failed.

[0014] Furthermore, in a mode of the present invention, the reception unit receives encoded video data from the communication partner apparatus, the communication apparatus further includes an output unit that outputs the video data to a decode unit, and for part of the video data failed to receive, the output unit outputs the video data supplemented with predetermined data to the decode unit.

[0015] Furthermore, in a mode of the present invention, the reception unit receives a plurality of packets provided with sequence numbers in one communication, and when the transmission unit successfully receives a packet with the last sequence number among the plurality of packets, the transmission unit transmits the size data in response to the reception of the plurality of packets even in a case where the reception of at least one of the plurality of packets has failed.

[0016] Furthermore, the present invention provides a generated data size control method including a step of transmitting data generated by a data generation unit to a communication partner apparatus, a step of receiving data transmitted from the communication partner apparatus, the data including size data indicating a size of the data successfully received by the communication partner apparatus, a step of specifying time it took for a round-trip communication in half-duplex wireless communication with the communication partner apparatus, and a step of controlling a size of the data generated by the data generation unit per unit time based on the specified time, the size indicated in the size data, and a given target time regarding the round-trip communication.

[0017] Furthermore, the present invention provides a communication method including a step of receiving data from a communication partner apparatus, and a step of transmitting size data indicating a size of successfully received data to the communication partner apparatus, in which in the step of transmitting the size data, the size data is transmitted even in a case where the reception of part of the data transmitted by the communication partner apparatus has failed.

[0018] Furthermore, the present invention provides a program causing a computer to execute a procedure of transmitting data generated by a data generation unit to a communication partner apparatus, a procedure of receiving data transmitted from the communication partner apparatus, the data including size data indicating a size of the data successfully received by the communication partner apparatus, a procedure of specifying time it took for a round-trip communication in half-duplex wireless communication with the communication partner apparatus, and a procedure of controlling a size of the data generated by the data generation unit per unit time based on the specified time, the size indicated in the size data, and a given target time regarding the round-trip communication.

[0019] Furthermore, the present invention provides another program including a procedure of receiving data from a communication partner apparatus, and a procedure of transmitting size data indicating a size of successfully received data to the communication partner apparatus, in which in the procedure of transmitting the size data, the size data is transmitted even in a case where the reception of part of the data transmitted by the communication partner apparatus has failed.

BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1 is a diagram illustrating an example of an overall configuration of an entertainment system according to an embodiment of the present invention.

[0021] FIG. 2A is a diagram illustrating an example of a configuration of a head-mounted display according to the embodiment of the present invention.

[0022] FIG. 2B is a diagram illustrating an example of a configuration of an entertainment apparatus according to the embodiment of the present invention.

[0023] FIG. 2C is a diagram illustrating an example of a configuration of a relay apparatus according to the embodiment of the present invention.

[0024] FIG. 3 is a diagram schematically illustrating an example of a state of half-duplex wireless communication between the relay apparatus and the HMD.

[0025] FIG. 4 is a diagram illustrating an example of a data structure of transaction data.

[0026] FIG. 5 is a diagram illustrating an example of a relationship between video data and payload data corresponding to the video data.

[0027] FIG. 6 is a diagram illustrating an example of change in time T_RT specified in each round-trip communication.

[0028] FIG. 7 is a functional block diagram illustrating an example of functions implemented in the head-mounted display, the entertainment apparatus, and the relay apparatus according to the embodiment of the present invention.

[0029] FIG. 8 is a flow chart illustrating an example of a flow of a process executed in the relay apparatus and the head-mounted display according to the embodiment of the present invention.

[0030] FIG. 9 is a flow chart illustrating an example of a flow of a process executed in the relay apparatus according to the embodiment of the present invention.

[0031] FIG. 10 is a functional block diagram illustrating an example of functions implemented in the head-mounted display, the entertainment apparatus, and the relay apparatus according to another embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

[0032] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0033] FIG. 1 is a diagram illustrating an example of an overall configuration of an entertainment system 10 according to the embodiment of the present invention. FIG. 2A is a diagram illustrating an example of a configuration of a head-mounted display (HMD) 12 according to the present embodiment. FIG. 2B is a diagram illustrating an example of a configuration of an entertainment apparatus 14 according to the present embodiment. FIG. 2C is a diagram illustrating an example of a configuration of a relay apparatus 16 according to the present embodiment.

[0034] As illustrated in FIG. 1, the entertainment system 10 according to the present embodiment includes the HMD 12, the entertainment apparatus 14, the relay apparatus 16, a display 18, a camera microphone unit 20, and a controller 22.

[0035] As illustrated in FIG. 2A, the HMD 12 according to the present embodiment includes, for example, a processor 30, a storage unit 32, a communication unit 34, an input/output unit 36, a display unit 38, a sensor unit 40, a voice output unit 42, and camera units 44.

[0036] The processor 30 is, for example, a program control device, such as a microprocessor, that operates according to a program installed on the HMD 12. Note that in place of the processor 30, a control circuit in which the process executed by the processor 30 can be implemented may be included in the HMD 12.

[0037] The storage unit 32 is, for example, a storage element or the like, such as a read only memory (ROM) and a random access memory (RAM). A program and the like executed by the processor 30 are stored in the storage unit 32.

[0038] The communication unit 34 is, for example, a communication interface, such as a wireless local area network (LAN) module. In the present embodiment, the communication unit 34 is arranged above and in front of the HMD 12 as illustrated in FIG. 1.

[0039] The input/output unit 36 is, for example, an input/output port, such as an HDMI (registered trademark) (High-Definition Multimedia Interface) port, a universal serial bus (USB) port, and an auxiliary (AUX) port.

[0040] The display unit 38 is, for example, a display, such as a liquid crystal display and an organic EL display, arranged in front of the HMD 12, and the display unit 38 displays a video and the like generated by the entertainment apparatus 14. In addition, the display unit 38 is placed in a housing of the HMD 12. The display unit 38 may be configured to, for example, receive a video signal output by the entertainment apparatus 14 and relayed by the relay apparatus 16 and output a video indicated by the video signal. The display unit 38 according to the present embodiment can, for example, display an image for left eye and an image for right eye to display a three-dimensional image. Note that the display unit 38 may not be able to display a three-dimensional image and may be able to display only a two-dimensional image.

[0041] The sensor unit 40 is, for example, a sensor, such as a motion sensor, that can detect the acceleration and the angular velocity. The sensor unit 40 outputs detection results, such as an amount of rotation and an amount of movement of the HMD 12, to the processor 30 at a predetermined sampling rate.

[0042] The voice output unit 42 is, for example, a speaker, such as a headphone and an earphone, arranged near the ears of the user, and the voice output unit 42 outputs voice and the like indicated by voice data generated by the entertainment apparatus 14. The voice output unit 42 receives, for example, a voice signal output by the entertainment apparatus 14 and relayed by the relay apparatus 16 and outputs voice indicated by the voice signal.

[0043] The camera unit 44 is, for example, a camera such as a digital camera, and the camera unit 44 images the surroundings of the user wearing the HMD 12 at a predetermined frame rate. As illustrated in FIG. 1, two camera units 44 are arranged above the display unit 38 in the present embodiment to allow imaging the front side of the display unit 38. Therefore, the camera units 44 according to the present embodiment can image the front side of the user wearing the HMD 12. In addition, the camera unit 44 according to the present embodiment is, for example, a stereo camera including a lens for capturing an image for left eye and a lens for generating an image for right eye.

[0044] The entertainment apparatus 14 according to the present embodiment is, for example, a computer, such as a game console, a digital versatile disc (DVD) player, and a Blu-ray (registered trademark) player. The entertainment apparatus 14 according to the present embodiment generates a video or voice by, for example, executing a stored game program or reproducing content recorded in an optical disk. Furthermore, the entertainment apparatus 14 according to the present embodiment outputs the video signal representing the video to be generated or the voice signal representing the voice to be generated to the HMD 12 or the display 18 through the relay apparatus 16.

[0045] The entertainment apparatus 14 according to the present embodiment includes, for example, a processor 50, a storage unit 52, a communication unit 54, and an input/output unit 56 as illustrated in FIG. 2B.

[0046] The processor 50 is, for example, a program control device, such as a CPU, that operates according to a program installed on the entertainment apparatus 14. The processor 50 according to the present embodiment also includes a GPU (Graphics Processing Unit) that draws an image in a frame buffer based on a graphics command or data supplied from the CPU. Note that in place of the processor 50, a control circuit in which the process executed by the processor 50 can be implemented may be included in the entertainment apparatus 14.

[0047] The storage unit 52 is, for example, a storage element, such as a ROM and a RAM, a hard disk drive, or the like. A program and the like executed by the processor 50 are stored in the storage unit 52. In addition, a region of the frame buffer in which the GPU draws an image is reserved in the storage unit 52 according to the present embodiment.

[0048] The communication unit 54 is, for example, a communication interface such as a wireless LAN module.

[0049] The input/output unit 56 is an input/output port, such as an HDMI port and a USB port.

[0050] The relay apparatus 16 according to the present embodiment is a computer that relays the video signal or the voice signal output from the entertainment apparatus 14 and outputs the video signal or the voice signal to the HMD 12 or the display 18.

[0051] The relay apparatus 16 according to the present embodiment includes, for example, a processor 60, a storage unit 62, a communication unit 64, and an input/output unit 66 as illustrated in FIG. 2C.

[0052] The processor 60 is, for example, a program control device, such as a CPU, that operates according to a program installed on the relay apparatus 16. Note that in place of the processor 60, a control circuit in which the process executed by the processor 60 can be implemented may be included in the relay apparatus 16.

[0053] The storage unit 62 is, for example, a storage element or the like, such as a ROM and a RAM. A program and the like executed by the processor 60 are stored in the storage unit 62.

[0054] The communication unit 64 is, for example, a communication interface such as a wireless LAN module. In the present embodiment, the communication unit 64 is included in front of the relay apparatus 16 as illustrated in FIG. 1.

[0055] The input/output unit 66 is an input/output port, such as an HDMI port and a USB port.

[0056] The display 18 according to the present embodiment is, for example, a liquid crystal display or the like, and the display 18 displays a video or the like indicated by the video signal output from the entertainment apparatus 14.

[0057] The camera microphone unit 20 according to the present embodiment includes, for example, a camera 20a that outputs data, such as a captured image of a subject, indicating the surroundings of the camera microphone unit 20 to the entertainment apparatus 14. In addition, the camera 20a according to the present embodiment is a stereo camera. In addition, the camera microphone unit 20 according to the present embodiment includes a microphone 20b that acquires voice of the surroundings and outputs voice data of the voice to the entertainment apparatus 14 after converting the voice into the voice data.

[0058] The HMD 12 and the relay apparatus 16 can, for example, mutually transmit and receive data through wireless communication. The entertainment apparatus 14 and the relay apparatus 16 are connected through, for example, an HDMI cable, a USB cable, or the like. The relay apparatus 16 and the display 18 are connected through, for example, an HDMI cable or the like. The entertainment apparatus 14 and the camera microphone unit 20 are connected through, for example, an AUX cable or the like.

[0059] The controller 22 according to the present embodiment is an operation input apparatus for inputting operations to the entertainment apparatus 14. The user can press a direction key or a button included in the controller 22 or tilt an operation stick to thereby use the controller 22 to input various operations. In addition, the controller 22 outputs input data associated with the input of operation to the entertainment apparatus 14 in the present embodiment. In addition, the controller 22 according to the present embodiment includes a USB port. Furthermore, the controller 22 can be connected to the entertainment apparatus 14 through a USB cable to output the input data to the entertainment apparatus 14 by wire. In addition, the controller 22 according to the present embodiment includes a wireless communication module or the like, and the controller 22 can also wirelessly output the input data to the entertainment apparatus 14.

[0060] In the present embodiment, the entertainment apparatus 14 executes, for example, a program such as a game program. Furthermore, the entertainment apparatus 14 generates a video, such as a video of a three-dimensional virtual space, corresponding to the execution status of the program. The video represents, for example, a view in a gaze direction from a point of view arranged in the virtual space. Furthermore, video data representing the video generated by the entertainment apparatus 14 is transmitted from the entertainment apparatus 14 to the HMD 12 through the relay apparatus 16.

[0061] In addition, sensing data is generated in the HMD 12 in the present embodiment. Here, examples of the sensing data include data indicating detection results of the acceleration and the angular velocity detected by the sensor unit 40 and images captured by the camera units 44. Furthermore, the sensing data generated by the HMD 12 is transmitted to the entertainment apparatus 14 through the relay apparatus 16.

[0062] The entertainment apparatus 14 generates a video corresponding to the sensing data received from the HMD 12. The entertainment apparatus 14 changes, for example, the position of the point of view arranged in the virtual space and the gaze direction according to the sensing data received from the HMD 12. Furthermore, the entertainment apparatus 14 generates a video representing a view in the changed gaze direction from the position of the changed point of view and transmits the video to the HMD 12. Furthermore, the HMD 12 that has received the video causes the display unit 38 to display the video. In this way, the video displayed on the display unit 38 is changed according to the movement of the head of the user in the present embodiment.

[0063] In the present embodiment, the relay apparatus 16 and the HMD 12 perform, for example, half-duplex wireless communication compliant with a standard such as, for example, Wi-Fi (registered trademark). FIG. 3 is a diagram schematically illustrating an example of a state of the half-duplex wireless communication between the relay apparatus 16 and the HMD 12.

[0064] As illustrated in FIG. 3, for example, transmission of video data from the relay apparatus 16 to the HMD 12 and transmission of sensing data from the HMD 12 to the relay apparatus 16 are alternately performed in the present embodiment. Hereinafter, a period in which the data is transmitted from the relay apparatus 16 to the HMD 12 will be referred to as a first period, and a period in which the data is transmitted from the HMD 12 to the relay apparatus 16 will be referred to as a second period. That is, the first period and the second period alternately come in the half-duplex wireless communication according to the present embodiment.

[0065] FIG. 4 is a diagram illustrating an example of a data structure of transaction data 70 according to the present embodiment. For example, the transaction data 70 transmitted from the relay apparatus 16 to the HMD 12 in one first period includes video data transmitted from the relay apparatus 16 to the HMD 12 in the first period. In addition, for example, the transaction data 70 transmitted from the HMD 12 to the relay apparatus 16 in one second period includes sensing data transmitted from the HMD 12 to the relay apparatus 16 in the second period.

[0066] As illustrated in FIG. 4, one piece of transaction data 70 includes one packet header 72 (expressed as PH in FIG. 4) and a plurality of divided packets 74. FIG. 4 illustrates that the transaction data 70 includes (N+1) divided packets 74. Furthermore, one divided packet 74 includes one piece of fragment information 76 (expressed as FI (0) or the like in FIG. 4) and one piece of payload data 78 (expressed as Payload (0) or the like in FIG. 4).

[0067] The packet header 72 of the transaction data 70 transmitted to the HMD 12 by the relay apparatus 16 includes, for example, a value of a bitrate that can be transmitted to the relay apparatus 16 by the HMD 12 (value PayloadBitrate_HtoP described later).

[0068] In addition, the packet header 72 of the transaction data 70 transmitted to the relay apparatus 16 by the HMD 12 includes, for example, a value indicating the size of data successfully received by the HMD 12 (value TransSize_PtoH_Act described later).

[0069] In addition, the packet header 72 also includes, for example, a sequence number of the last divided packet 74.

[0070] The fragment information 76 included in the divided packet 74 includes, for example, a sequence number of the divided packet 74 and the sequence number of the last divided packet 74 in the transaction data 70 including the divided packet 74.

[0071] The payload data 78 corresponds to, for example, the video data transmitted from the relay apparatus 16 to the HMD 12 or the sensing data transmitted from the HMD 12 to the relay apparatus 16.

[0072] FIG. 5 is a diagram illustrating an example of a relationship between the video data transmitted from the relay apparatus 16 to the HMD 12 and the payload data 78 corresponding to the video data.

[0073] Here, the video data according to the present embodiment is encoded according to, for example, a standard of moving picture experts group (MPEG). Furthermore, the encoded video data is divided into packetized elementary streams (Packetized Elementary Stream: PES) that are variable-length packets. FIG. 5 illustrates packets of four PESs (PES(0) to PES(3)).

[0074] Furthermore, in the present embodiment, the PES is further divided into, for example, transport packets (TS), each with a fixed length of 188 bytes including a header of 4 bytes and a payload of 184 bytes.

[0075] FIG. 5 illustrates twenty TSs (TS(0) to TS(19)). The example in FIG. 5 illustrates that six TSs (TS(0) to TS(5)) are generated based on PES(0). In addition, the example illustrates that six TSs (TS(6) to TS(11)) are generated based on PES(1). In addition, the example illustrates that five TSs (TS(12) to TS(16)) are generated based on PES(2). In addition, the example illustrates that three TSs (TS(17) to TS(19)) are generated based on PES(3). Here, for a TS including a part without a corresponding PES, such as TS(11), TS(16), and TS(19), the part is padded with a predetermined value.

[0076] Pieces of payload data 78 are then generated, each corresponding to one or a plurality of TSs. The example in FIG. 5 illustrates five pieces of payload data 78 (Payload(0) to Payload(4)). The example in FIG. 5 illustrates that Payload(0) corresponds to TS(0) to TS(3). In addition, the example illustrates that Payload(1) corresponds to TS(4) to TS(7). In addition, the example illustrates that Payload(2) corresponds to TS(8) to TS(11). In addition, the example illustrates that Payload(3) corresponds to TS(12) to TS(15). In addition, the example illustrates that Payload(4) corresponds to TS(16) to TS(19). Here, the payload data 78 is always aligned with the TSs, and there is no case in which one TS is separately allocated to a plurality of pieces of payload data 78.

[0077] As illustrated in FIG. 3, the relay apparatus 16 transmits the packet header 72 to the HMD 12 and then transmits the rest of the transaction data 70 to the HMD 12 in one first period. Furthermore, in response to the reception of the transaction data 70, the HMD 12 transmits the packet header 72 to the relay apparatus 16 and then transmits the rest of the transaction data 70 to the relay apparatus 16 in one second period following the first period. Note that the relay apparatus 16 and the HMD 12 may perform communication of data other than the transaction data 70, such as voice data and a message for control.

[0078] Furthermore, in the present embodiment, the relay apparatus 16 can specify the time it took for a round-trip communication (round trip time) from the transmission of the packet header 72 of the video data to the reception of all of the sensing data transmitted in response to the reception of the video data as illustrated in FIG. 3. Hereinafter, the time will be expressed as time T_RT. Here, the time T_RT corresponds to the time it took for a round-trip communication corresponding to the time from the start of the transmission of the video data in one first period to the end of the reception of the sensing data in the second period following the first period.

[0079] Furthermore, in the present embodiment, the relay apparatus 16 can specify the time from the transmission of the packet header 72 of the video data to the reception of the packet header 72 of the sensing data transmitted from the HMD 12 in response to the end of the reception of the video data as illustrated in FIG. 3. The time substantially corresponds to the time it took for the transmission of the video data from the relay apparatus 16 to the HMD 12. Hereinafter, the time will be expressed as time T_ph. In addition, the time obtained by subtracting the time T_ph from the time T_RT substantially corresponds to the time it took for the transmission of the sensing data from the HMD 12 to the relay apparatus 16. Hereinafter, the time will be expressed as time T_hp.

[0080] Furthermore, a given target time regarding a round-trip communication in the half-duplex wireless communication is set in advance in the present embodiment. Hereinafter, the target time will be expressed as time TargetTransferTime.

[0081] It is desirable that the quality of the data used in the HMD 12 be as high as possible like video data according to the present embodiment. Therefore, it is desirable that the size of the data generated by the entertainment apparatus 14 per unit time be as large as possible. On the other hand, it is important to bring the round trip time closer to the time TargetTransferTime even by reducing the size of the data generated per unit time in order to reduce the discomfort of the user in the communication that requires promptness as in the present embodiment. For example, it is important to reduce the latency of each frame image included in the video displayed by the HMD 12.

[0082] Considering the above, the size of the data generated by the entertainment apparatus 14 per unit time is controlled as follows in the present embodiment so that the time T_RT becomes equal to or smaller than the time TargetTransferTime as much as possible.

[0083] Furthermore, in a case where, for example, the communication environment is poor, part of the data may be missing, and the missing part may not reach the communication partner. For example, one or a plurality of divided packets 74 may not reach the communication partner.

[0084] In such a case, the time it takes for a round-trip communication may not be close to the given target time unless the size of the data successfully communicated by the communication partner is taken into account to control the size of the data generated per unit time.

[0085] Therefore, in the present embodiment, the size of the data generated per unit time is controlled as follows based on the size of the data successfully communicated by the communication partner.

[0086] In addition, when part of the data transmitted from the relay apparatus 16 to the HMD 12 is missing, the relay apparatus 16 cannot calculate the time T_RT unless the sensing data is transmitted from the HMD 12 to the relay apparatus 16. As a result, the size of the data to be generated by the entertainment apparatus 14 per unit time cannot be calculated.

[0087] Therefore, in the present embodiment, the sensing data is transmitted from the HMD 12 to the relay apparatus 16 as follows even when part of the data transmitted from the relay apparatus 16 to the HMD 12 is missing.

[0088] In addition, when, for example, the communication environment is deteriorated, a communication path search process based on beamforming or the like may occur, and the communication environment may become unstable. FIG. 6 is a diagram illustrating an example of change in the time T_RT specified in each round-trip communication. In the example of FIG. 6, beamforming is intermittently and continuously occurring in the communication path search process. In the example of FIG. 6, the time T_RT significantly fluctuates in a range of time from approximately 2000 microseconds to 5500 microseconds. Under the circumstances, even if, for example, the size of the data generated per unit time is controlled to be large based on the fact that the time T_RT is shorter than the target time, the communication of a large size of data may not be possible in the next round-trip communication. Therefore, in such a situation, it is desirable not to significantly increase the size of the data generated per unit time even in the case where the time T_RT is shorter than the target time.

[0089] Considering the above, in the present embodiment, the magnitude of change in the value corresponding to the size of the data generated per unit time corresponds to the result of estimation of whether or not the communication path search process is occurring as described below.

[0090] Conversely, in the situation described above, communication of data in a shorter time than the target time may be possible in the next round-trip communication when, for example, the size of the data generated per unit time is controlled to be small based on the fact that the time T_RT is longer than the target time. If the next round-trip communication is immediately started in such a case, the video data may not be generated in time, and empty video data may be transmitted from the relay apparatus 16 to the HMD 12. The transmission of empty data is a waste of power consumption.

[0091] Considering the above, the present embodiment can suppress the waste of power consumption as follows.

[0092] Here, the functions of the entertainment system 10 according to the present embodiment and the process executed by the entertainment system 10 according to the present embodiment will be further described, mostly in relation to the matters described above.

[0093] FIG. 7 is a functional block diagram illustrating an example of functions implemented in the HMD 12, the entertainment apparatus 14, and the relay apparatus 16 according to the present embodiment. Note that not all of the functions illustrated in FIG. 7 need to be implemented in the entertainment system 10 according to the present embodiment, and functions other than the functions illustrated in FIG. 7 may also be implemented.

[0094] As illustrated in FIG. 7, the entertainment apparatus 14 according to the present embodiment has functions including, for example, a video data generation unit 80, a video data transmission unit 82, a generation control reception unit 84, and a sensing data reception unit 86. The master unit of the video data generation unit 80 is the processor 50. The master unit of the video data transmission unit 82, the generation control reception unit 84, and the sensing data reception unit 86 is the input/output unit 56.

[0095] A program that is installed on the entertainment apparatus 14 as a computer and that includes instructions corresponding to the functions may be executed by the processor 50 to implement the functions. The program may be supplied to the entertainment apparatus 14 through a computer-readable information storage medium, such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, and a flash memory, or through the Internet or the like.

[0096] In addition, as illustrated in FIG. 7, the relay apparatus 16 according to the present embodiment has functions including, for example, a target time information storage unit 90, a video data reception unit 92, a video data buffer 94, a video data transmission unit 96, a sensing data reception unit 98, a sensing data transmission unit 100, a required time specification unit 102, a search occurrence estimation unit 104, a bitrate determination unit 106, a data size determination unit 108, and a generation control unit 110. The master unit of the target time information storage unit 90 and the video data buffer 94 is the storage unit 62. The master unit of the video data reception unit 92 and the sensing data transmission unit 100 is the input/output unit 66. The master unit of the video data transmission unit 96 and the sensing data reception unit 98 is the communication unit 64. The master unit of the required time specification unit 102, the search occurrence estimation unit 104, the bitrate determination unit 106, and the data size determination unit 108 is the processor 60. The master units of the generation control unit 110 are the processor 60 and the input/output unit 66.

[0097] A program that is installed on the relay apparatus 16 as a computer and that includes instructions corresponding to the functions may be executed by the processor 60 to implement the functions. The program may be supplied to the relay apparatus 16 through a computer-readable information storage medium, such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, and a flash memory, or through the Internet or the like.

[0098] In addition, as illustrated in FIG. 7, the HMD 12 according to the present embodiment has functions including, for example, a video data reception unit 120, a video data output unit 122, a decode unit 124, a display control unit 126, a sensing data generation unit 128, a sensing data buffer 130, and a sensing data transmission unit 132. The master unit of the video data reception unit 120 and the sensing data transmission unit 132 is the communication unit 34. The master unit of the video data output unit 122 and the decode unit 124 is the processor 30. The master units of the display control unit 126 are the processor 30 and the display unit 38. The master units of the sensing data generation unit 128 are the processor 30, the sensor unit 40, and the camera units 44. The master unit of the sensing data buffer 130 is the storage unit 32.

[0099] A program that is installed on the HMD 12 as a computer and that includes instructions corresponding to the functions may be executed by the processor 30 to implement the functions. The program may be supplied to the HMD 12 through a computer-readable information storage medium, such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, and a flash memory, or through the Internet or the like.

[0100] The video data generation unit 80 generates, for example, video data representing the video to be displayed on the display unit 38 of the HMD 12 in the present embodiment.

[0101] The video data transmission unit 82 transmits, for example, the video data generated by the video data generation unit 80 to the relay apparatus 16 in the present embodiment.

[0102] The generation control reception unit 84 receives, for example, generation control information transmitted by the relay apparatus 16 in the present embodiment. Here, the generation control information is, for example, information indicating the size of the data to be generated per unit time. The video data generation unit 80 is controlled to generate the data of the size indicated in the generation control information per unit time in response to the reception of the generation control information by the generation control reception unit 84.

[0103] The sensing data reception unit 86 receives, for example, sensing data, such as data indicating detection results of the sensor unit 40 of the HMD 12 and images captured by the camera units 44, from the relay apparatus 16 in the present embodiment. In the present embodiment, the video data generation unit 80 generates video data corresponding to the sensing data received by the sensing data reception unit 86.

[0104] The target time information storage unit 90 stores, for example, target time information in which the value of the target time TargetTransferTime is set in the present embodiment.

[0105] The video data reception unit 92 receives, for example, video data transmitted from the entertainment apparatus 14 in the present embodiment.

[0106] The video data buffer 94 stores, for example, the video data received by the video data reception unit 92 in the present embodiment.

[0107] The video data transmission unit 96 transmits, for example, the video data generated by the video data generation unit 80 to the HMD 12 in the present embodiment. For example, the video data transmission unit 96 transmits the video data stored in the video data buffer 94 to the HMD 12.

[0108] The sensing data reception unit 98 receives, for example, sensing data transmitted by the HMD 12 in the present embodiment.

[0109] The sensing data transmission unit 100 transmits, for example, the sensing data received by the sensing data reception unit 98 to the entertainment apparatus 14 in the present embodiment.

[0110] The required time specification unit 102 specifies, for example, the time T_RT it took for a round-trip communication in the half-duplex wireless communication between the relay apparatus 16 and the HMD 12 in response to the end of the round-trip communication in the present embodiment.

[0111] The search occurrence estimation unit 104 estimates whether or not a communication path search process is occurring between the relay apparatus 16 and the HMD 12. Here, the communication path search process may include continuously occurring beamforming.

[0112] In addition, the search occurrence estimation unit 104 may also determine an evaluation value regarding whether or not the communication path search process is occurring based on the length of the time T_RT. Here, an example of the evaluation value includes a value obtained by subtracting a given value THR_INT described later from the value indicating the time T_RT.

[0113] Note that the details of the estimation by the search occurrence estimation unit 104 will be described later.

[0114] The bitrate determination unit 106 determines, for example, a bitrate that can be transmitted to the HMD 12 by the relay apparatus 16 in a later round-trip communication in the present embodiment. Here, the bitrate may be determined based on, for example, the specified time T_RT, the size of the data communicated with the HMD 12 in a round-trip communication, and the given target time TargetTransferTime regarding a round-trip communication. In addition, the bitrate determination unit 106 may determine a bitrate that can be transmitted to the relay apparatus 16 by the HMD 12.

……
……
……