Sony Patent | Information processing system, information processing method, and program

Patent: Information processing system, information processing method, and program

Drawings: Click to check drawins

Publication Number: 20210081092

Publication Date: 20210318

Applicant: Sony

Assignee: Sony Corporation

Abstract

There is provided an information processing system, an information processing method, and a program capable of adaptively controlling a display on a display surface in response to an operation of moving an object in a certain display surface with respect to a transfer object. The information processing system includes a processing unit that performs, on a first display surface, a predetermined display control indicating that one or more pieces of display information in a second display surface corresponding to the first display surface is changed to be operable in the first display surface when a first operation for a first user to move a first object to a detection area having a predetermined positional relationship with one or more predetermined transfer objects positioned in the first display surface, and changes a display state of the second display surface based on a detection result of a second operation by the first user after the first operation.

Claims

1. An information processing system, comprising: a processing unit that performs, on a first display surface, a predetermined display control indicating that one or more pieces of display information in a second display surface corresponding to the first display surface is changed to be operable in the first display surface when a first operation for a first user to move a first object to a detection area having a predetermined positional relationship with one or more predetermined transfer objects positioned in the first display surface, and changes a display state of the second display surface based on a detection result of a second operation by the first user after the first operation.

2. The information processing system according to claim 1, wherein the processing unit displays information corresponding to the first object on the second display surface based on the detection result of the second operation.

3. The information processing system according to claim 2, wherein the second operation is an operation that continuously performs the first operation while maintaining an operation state for the first object in a predetermined operation state.

4. The information processing system according to claim 3, wherein the predetermined display control includes displaying, on the first display surface, one or more pieces of display information within a first target area on the second display surface.

5. The information processing system according to claim 4, wherein the first target area is a partial area in the second display surface, the processing unit further changes the first target area from a first area in the second display surface to a second area in the second display surface based on the detection result of the second operation, and the processing unit switches display information displayed on the first display surface from one or more pieces of display information within the first area to the one or more pieces of display information within the second area when the first target area is changed from the first area to the second area.

6. The information processing system according to claim 4, wherein the first target area is substantially the entirety of the second display surface, the predetermined display control includes displaying, on the first display surface, a map image indicating one or more pieces of display information displayed on substantially the entirety of the second display surface, and a size of the map image is smaller than a size of the first display surface.

7. The information processing system according to claim 4, wherein a location where the second display surface is positioned is remote from a location where the first display surface is positioned, the processing unit displays the one or more pieces of display information within the second target area on the second display surface on a third display surface positioned at a location remote from each of the locations where the first display surface is positioned and the location where the second display surface is positioned, and the processing unit further displays a display indicating a positional relationship between the first target area and the second target area on the first display surface while the one or more pieces of display information in the first target area are displayed on the first display surface.

8. The information processing system according to claim 7, further comprising: a receiving unit that receives pointing information indicating a position where a second user is pointing on the third display surface while the one or more pieces of display information in the second target area is displayed on the third display surface, wherein the processing unit further displays a predetermined image at a position in the first display surface corresponding to a pointing position indicated by the pointing information while the one or more pieces of display information in the first target area is displayed on the first display surface.

9. The information processing system according to claim 4, wherein the processing unit further determines a transfer position on the second display surface of the information corresponding to the first object based on the detection result of the second operation.

10. The information processing system according to claim 9, wherein the first target area is substantially the entirety of the second display surface, the predetermined display control includes displaying, on the first display surface, a map image indicating one or more pieces of display information displayed on substantially the entirety of the second display surface, and a size of a map image is smaller than a size of the first display surface, and the second operation includes specifying the transfer position of the information corresponding to the first object on the second display surface by the first user in the map image.

11. The information processing system according to claim 9, wherein the information corresponding to the first object is an image corresponding to the first object, and the processing unit further displays the image corresponding to the first object to be displayed at the transfer position on the second display surface.

12. The information processing system according to claim 3, wherein the processing unit further determines at least one display surface of the transfer destination of the information corresponding to the first object from among a plurality of display surfaces corresponding to the first display surface based on the information indicating a state of the one or more predetermined transfer objects and the detection result of the first operation, and the plurality of display surfaces include the second display surface.

13. The information processing system according to claim 12, wherein the one or more predetermined transfer objects exist in plurality, each of the plurality of predetermined transfer objects is a real object, and the processing unit determines at least one display surface of the transfer destination of the information corresponding to the first object from among the plurality of display surfaces based on whether or not a distance between any two transfer objects of the plurality of predetermined transfer objects is equal to or less than a predetermined threshold and whether or not an arrangement of the any two transfer objects is a predetermined arrangement, when the first operation is detected.

14. The information processing system according to claim 12, wherein the predetermined transfer object is a real object having a plurality of surfaces, each of the plurality of surfaces of the predetermined transfer object is associated with a different display surface of the plurality of display surfaces, the first operation includes allowing the first user to arrange the first object on one of the plurality of surfaces of the predetermined transfer object, and the processing unit determines the display surface associated with the surface on which the first object is arranged among the plurality of surfaces of the predetermined transfer object as the display surface of the transfer destination of the information corresponding to the first object, when the first operation is detected.

15. The information processing system according to claim 12, wherein the predetermined transfer object is a real object having at least one inclined surface, the first operation includes allowing the first user to arrange the first object on any of the at least one inclined surface of the predetermined transfer object, and the processing unit determines the display surface positioned in a direction in which the first object is directed as the display surface of the transfer destination of the information corresponding to the first object from among the plurality of display surfaces, when the first operation is detected.

16. The information processing system according to claim 12, wherein the predetermined transfer object is a real object, and the processing unit continuously changes a display position of the image within the second display surface based on a detection result of a moving progress of the first object with respect to the predetermined transfer object and a detection result of a posture of the predetermined transfer object during the first operation, when the image corresponding to the first object is displayed on the second display surface.

17. The information processing system according to claim 12, wherein the information corresponding to the first object is an image corresponding to the first object, the information associated with the second display surface includes conversion rule information of a character string associated with the second display surface, and the processing unit displays an image including a second character string, which is associated with the second display surface and converted based on the conversion rule information of the character string and the first character string as the image corresponding to the first object on the second display surface when the first character string is included in the first object.

18. The information processing system according to claim 12, wherein the information corresponding to the first object is an image corresponding to the first object, and the information associated with the second display surface includes conversion rule information of a format of an image associated with the second display surface, and the processing unit displays an image of a format determined based on the conversion rule information of the format of the image associated with the second display surface and the first object as the image corresponding to the first object on the second display surface.

19. An information processing method, comprising: performing, on a first display surface, a predetermined display control indicating that one or more pieces of display information in a second display surface corresponding to the first display surface is changed to be operable in the first display surface when a first operation for a first user to move a first object to a detection area having a predetermined positional relationship with one or more predetermined transfer objects positioned in the first display surface; and changing, by a processor, a display state of the second display surface based on a detection result of a second operation by the first user after the detection of the first operation.

20. A program for executing a computer to function as a processing unit that performs, on a first display surface, a predetermined display control indicating that one or more pieces of display information in a second display surface corresponding to the first display surface is changed to be operable in the first display surface when a first operation for a first user to move a first object to a detection area having a predetermined positional relationship with one or more predetermined transfer objects positioned in the first display surface, and changes a display state of the second display surface based on a detection result of a second operation by the first user after the detection of the first operation.

Description

FIELD

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

BACKGROUND

[0002] Conventionally, various technologies related to augmented reality (AR) have been developed. In AR, additional information associated with an object in an environment where a user is positioned can be presented to a user.

[0003] For example, Patent Literature 1 below describes a technology for detecting a marker by analyzing a captured image and calling a function associated with the detected marker.

CITATION LIST

Patent Literature

[0004] Patent Literature 1: JP 2015-90524** A**

SUMMARY

Technical Problem

[0005] However, in the technology described in Patent Literature 1, when an operation of moving an object in a certain display surface with respect to a transfer object is detected, it is not considered to adaptively control the display on the display surface in response to the operation.

[0006] Therefore, the present disclosure provides a new and improved information processing system, information processing method, and program capable of adaptively controlling a display on the display surface in response to an operation of moving an object in a certain display surface with respect to a transfer object.

Solution to Problem

[0007] According to the present disclosure, an information processing system is provided that includes: a processing unit that performs, on a first display surface, a predetermined display control indicating that one or more pieces of display information in a second display surface corresponding to the first display surface is changed to be operable in the first display surface when a first operation for a first user to move a first object to a detection area having a predetermined positional relationship with one or more predetermined transfer objects positioned in the first display surface, and changes a display state of the second display surface based on a detection result of a second operation by the first user after the first operation.

[0008] Moreover, according to the present disclosure, an information processing method is provided that includes: performing, on a first display surface, a predetermined display control indicating that one or more pieces of display information in a second display surface corresponding to the first display surface is changed to be operable in the first display surface when a first operation for a first user to move a first object to a detection area having a predetermined positional relationship with one or more predetermined transfer objects positioned in the first display surface; and changing, by a processor, a display state of the second display surface based on a detection result of a second operation by the first user after the detection of the first operation.

[0009] Moreover, according to the present disclosure, a program is provided that executes a computer to function as a processing unit that performs, on a first display surface, a predetermined display control indicating that one or more pieces of display information in a second display surface corresponding to the first display surface is changed to be operable in the first display surface when a first operation for a first user to move a first object to a detection area having a predetermined positional relationship with one or more predetermined transfer objects positioned in the first display surface, and changes a display state of the second display surface based on a detection result of a second operation by the first user after the detection of the first operation.

Advantageous Effects of Invention

[0010] As described above, according to the present disclosure, it is possible to adaptively control the display on the display surface in response to the operation of moving the object in a certain display surface with respect to the transfer object. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a diagram illustrating a configuration example of an information processing system 10 according to an embodiment of the present disclosure.

[0012] FIG. 2 is a diagram illustrating another configuration example of the information processing system 10 according to the embodiment.

[0013] FIG. 3 is a diagram illustrating still another configuration example of the information processing system 10 according to the embodiment.

[0014] FIG. 4 is a diagram illustrating yet another configuration example of the information processing system 10 according to the embodiment.

[0015] FIG. 5 is a diagram illustrating another configuration example of the system according to the embodiment.

[0016] FIG. 6 is a block diagram illustrating an example of a functional configuration of the information processing system 10 according to the embodiment.

[0017] FIG. 7 is a diagram illustrating an example of a transfer operation for transferring information corresponding to an object to another display surface according to the embodiment.

[0018] FIG. 8 is a diagram illustrating a configuration example of a surface attribute information DB 128 according to the embodiment.

[0019] FIG. 9A is a diagram illustrating an example of a function of a surface attribute matching processing unit 104 according to the embodiment.

[0020] FIG. 9B is a diagram illustrating an example of the function of the surface attribute matching processing unit 104 according to the embodiment.

[0021] FIG. 10A is a diagram illustrating another example of the function of the surface attribute matching processing unit 104 according to the embodiment.

[0022] FIG. 10B is a diagram illustrating still another example of the function of the surface attribute matching processing unit 104 according to the embodiment.

[0023] FIG. 11 is a flowchart illustrating a part of a processing flow according to the embodiment.

[0024] FIG. 12 is a flowchart illustrating a part of the processing flow according to the embodiment.

[0025] FIG. 13 is a diagram illustrating an example of transferring an image corresponding to an object according to a first application example of the present embodiment.

[0026] FIG. 14 is a diagram illustrating another example of transferring the image corresponding to the object according to the first application example.

[0027] FIG. 15 is a diagram illustrating an example of transferring an image corresponding to an object according to a second application example of the present embodiment.

[0028] FIG. 16 is a diagram illustrating another example of transferring the image corresponding to the object according to the second application example.

[0029] FIG. 17 is a diagram illustrating an example of transferring an image corresponding to an object according to a third application example of the present embodiment.

[0030] FIG. 18 is a diagram illustrating another example of transferring the image corresponding to the object according to the third application example.

[0031] FIG. 19A is a diagram illustrating an example of a transfer operation of an image corresponding to an object according to a fourth application example of the present embodiment.

[0032] FIG. 19B is a diagram illustrating an example of transferring an image corresponding to the object according to the fourth application example.

[0033] FIG. 20 is a diagram illustrating a configuration example of a surface attribute information DB 128 according to a fifth application example of the present embodiment.

[0034] FIG. 21A is a diagram illustrating an example of a transfer operation of an image corresponding to an object according to the fifth application example.

[0035] FIG. 21B is a diagram illustrating an example in which an image corresponding to an object to be transferred according to the fifth application example is converted and displayed according to a display surface of a transfer destination.

[0036] FIG. 22 is a diagram illustrating a configuration example of a system according to a sixth application example of the present embodiment.

[0037] FIG. 23 is a diagram illustrating a configuration example of a surface attribute information DB 128 according to the sixth application example.

[0038] FIG. 24A is a diagram illustrating an example of a transfer operation of an image corresponding to an object with respect to a display screen of a smartphone 50 according to the sixth application example.

[0039] FIG. 24B is a diagram illustrating an example in which an image corresponding to an object to be transferred according to the sixth application example is converted and displayed according to a display surface of a transfer destination.

[0040] FIG. 24C is a diagram illustrating an example in which the image corresponding to the object to be transferred according to the sixth application example is converted and displayed according to the display surface of the transfer destination.

[0041] FIG. 25 is a diagram illustrating a configuration example of a surface attribute information DB 128 according to a seventh application example of the present embodiment.

[0042] FIG. 26A is a diagram illustrating an example of a transfer operation of an image corresponding to an object according to the seventh application example.

[0043] FIG. 26B is a diagram illustrating an example in which an image corresponding to an object to be transferred according to the seventh application example is converted and displayed according to a display surface of a transfer destination.

[0044] FIG. 27A is a diagram illustrating an example of a transfer operation of an image corresponding to an object according to an eighth application example of the present embodiment.

[0045] FIG. 27B is a diagram illustrating an example of transferring an image corresponding to the object according to the eighth application example.

[0046] FIG. 27C is a diagram illustrating an example in which display information in a target area in the display surface of the transfer destination is displayed on a display surface of a transfer source according to the eighth application example.

[0047] FIG. 27D is a diagram illustrating a display example of the display surface of the transfer destination at timing illustrated in FIG. 27C.

[0048] FIG. 28A is a diagram illustrating an example of an operation of a display surface of a transfer source for scrolling a target area in a display surface of a transfer destination according to a ninth application example of the present embodiment.

[0049] FIG. 28B is a diagram illustrating an example of movement of the target area based on the operation illustrated in FIG. 28A.

[0050] FIG. 29A is a diagram illustrating an example of another operation of a display surface of a transfer source for scrolling a target area in a display surface of a transfer destination according to the ninth application example.

[0051] FIG. 29B is a diagram illustrating an example of the movement of the target area based on the operation illustrated in FIG. 28B.

[0052] FIG. 30A is a diagram illustrating a display example of a display surface of a transfer destination according to a tenth application example of the present embodiment.

[0053] FIG. 30B is a diagram illustrating an example in which a map image indicating substantially the entire display surface of the transfer destination at timing illustrated in FIG. 30A is displayed on the display surface of the transfer source.

[0054] FIG. 31A is a diagram illustrating a display example of a display surface of a transfer destination according to an eleventh application example of the present embodiment.

[0055] FIG. 31B is a diagram illustrating a display example of a display surface of a transfer source at timing illustrated in FIG. 31A according to the eleventh application example.

[0056] FIG. 32A is a diagram illustrating a display example of a whiteboard 20-2 when an editing task on a whiteboard 20-2 is simultaneously performed on a table surface 20-1 and a table surface 20-3, according to a twelfth application example of the present embodiment.

[0057] FIG. 32B is a diagram illustrating a display example of the table surface 20-1 at timing illustrated in FIG. 32A.

[0058] FIG. 32C is a diagram illustrating a display example of the table surface 20-3 at the timing illustrated in FIG. 32A.

[0059] FIG. 33A is a diagram illustrating an example of an operation of moving an object to be transferred into a transfer area according to a thirteenth application example of the present embodiment.

[0060] FIG. 33B is a diagram illustrating a state where a user continues to touch the object to be transferred after the timing illustrated in FIG. 33A.

[0061] FIG. 33C is a diagram illustrating a display example of a map image displayed after the timing illustrated in FIG. 33B for specifying a transfer position of the corresponding object.

[0062] FIG. 34 is a diagram illustrating an example of a hardware configuration of the information processing system 10 according to the present embodiment.

[0063] FIG. 35 is a diagram illustrating a configuration example of a system according to a modified example of the present embodiment.

DESCRIPTION OF EMBODIMENTS

[0064] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the present specification and drawings, components having substantially the same functional configuration will be denoted by the same reference numerals, and a redundant description thereof will be omitted.

[0065] In this specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numeral. For example, a plurality of components having substantially the same functional configuration are distinguished as necessary, such as a table 90a and a table 90b. However, when there is no need to particularly distinguish each of the plurality of components having substantially the same functional configuration, the components are denoted by only the same reference numeral. For example, when there is no need to particularly distinguish between the table 90a and the table 90b, the table 90a and the table 90b are simply referred to as a table 90.

[0066] In addition, the “embodiment for carrying out the invention” will be described in the order of items below.

[0067] 1. Configuration of Information Processing System

[0068] 2. Detailed Description of Embodiment

[0069] 3. Application Example

[0070] 4. Hardware Configuration

[0071] 5. Modified Example

1.* CONFIGURATION OF INFORMATION PROCESSING SYSTEM*

[0072] First, a configuration example of an information processing system according to an embodiment of the present disclosure will be described with reference to FIG. 1. Note that in this specification, the system can mean a configuration for executing predetermined processing. The system may be constituted by one device, or may be constituted by a plurality of devices. In addition, the information processing system 10 according to the present embodiment may also be configured to be able to execute predetermined processing as a whole, and may be arbitrary which configuration in the information processing system 10 is regarded as one device.

[0073] FIG. 1 is a diagram illustrating a configuration example of the information processing system 10. Referring to FIG. 1, an information processing system 10a according to the present embodiment includes an input unit 120a and an output unit 124a.

1-1. Output Unit 124

[0074] An output unit 124a can be configured to include an output device 162 described later. The output unit 124a displays various information on a table 90a. The output unit 124a can be a projection unit (projector). For example, as illustrated in FIG. 1, the output unit 124a can be arranged above the table 90a at a predetermined distance from the table 90a while being suspended from a ceiling. In this case, the output unit 124a can project information on a top surface of the table 90a. For example, the output unit 124a may be a pendant light or a desk stand light. Such a method of displaying information on a top surface of a table 90a from above is also referred to as “projection type”. In the following, the top surface of the table 90 may be referred to as display surface 20. The display surface 20 includes a surface (screen) to be projected by an output unit 124.

[0075] For example, the output unit 124a displays a virtual display object according to control of a display processing unit 122 described later. The display object is, for example, a window, a user interface (UI) object, or the like. The UI object is a predetermined image (still image or moving image) that receives various operations (selection, input, or the like) by a user. For example, the UI object is an image including graphical user interface (GUI) components (for example, a button, a slider, a check box, a text box, a software keyboard, and the like). In addition, the UI object can be arranged in the window.

1-2. Input Unit 120

[0076] An input unit 120a can be configured to include an input device 160 described later. The input unit 120a includes, for example, a camera that captures an image of the table 90a with one lens. Alternatively, the input unit 120a can include a stereo camera capable of recording information in a depth direction by capturing the image of the table 90a with two lenses. As the stereo camera, for example, a visible light camera, an invisible light camera capable of detecting invisible light such as infrared light, or the like can be used. In addition, the input unit 120a can further include a voice input device such as a microphone that collects a voice uttered by a user or environmental sound of the surrounding environment.

[0077] When the above-described camera is used as the input unit 120a, the information processing system 10a analyzes an image (captured image) captured by the camera, thereby detecting a position of an object (for example, a user’s hand or the like) positioned on the display surface 20. Note that in this specification and the drawings, the term “hand” can be mainly used as an expression meaning a hand itself. However, the present invention is not limited to such an example, and this description may be used as an expression meaning a finger, an arm, or two or more (for example, an upper limb or the like) of these parts.

[0078] In addition, when the stereo camera is used as the input unit 120a, the information processing system 10a analyzes the image captured by the stereo camera, thereby acquiring depth information of the object in addition to position information of an object positioned on the display surface 20. The information processing system 10a can detect touch or approach of the user’s hand to the display surface 20 and separation of the hand from the display surface 20, based on the depth information. Note that the input unit 120a may have another type of depth sensor (for example, a time-of-flight type sensor, a structured light type sensor, or the like) instead of the stereo camera. In this case, the depth sensor can acquire the depth information of the object positioned on the display surface 20.

[0079] In the present embodiment, a position of an operation tool (for example, a user’s hand, various operating members such as a stylus) on the display surface 20 can be detected based on an image captured by the input unit 120a, and various information can be input based on the detected position of the operation tool. That is, the user can perform various operation inputs by moving the operation tool on the display surface 20. For example, the touch of the user’s hand to the window or the UI object is detected, and as a result, the operation input to the window or the UI object is performed.

[0080] In addition, the camera included in the input unit 120a can not only photograph the top surface of the table 90a, but also photograph a user existing around the table 90a. In this case, the information processing system 10a can detect the position of the user around the table 90a based on the image captured by the input unit 120a. In addition, the information processing system 10a may perform personal recognition of a user by detecting physical characteristics (such as a size of a face or a body) of each user based on the captured image.

[0081] Note that an operation input of a user is not limited to the example described above, and may be executed by another method. For example, the input unit 120a may be arranged as a touch panel on the top surface (display surface 20a) of the table 90a, and the operation input of the user may also be detected by a touch of the user’s finger or the like on the touch panel. Alternatively, a gesture operation of the user may be photographed by a camera included in the input unit 120a, and the operation input of the user may be detected based on the photographed image.

1-3. Modified Example

[0082] The configuration of the information processing system 10a according to the present embodiment has been described above. Note that the configuration of the information processing system 10 according to the present embodiment is not limited to the example illustrated in FIG. 1. The information processing system 10 may have another configuration as illustrated in FIGS. 2 to 4, for example.

1-3-1. First Modified Example

[0083] FIG. 2 is a diagram illustrating another configuration example (information processing system 10b) of the information processing system 10 according to the present embodiment. As illustrated in FIG. 2, in the information processing system 10b, the output unit 124b is arranged below the table 90b. The output unit 124b is, for example, a projector, and projects information from below the table 90b toward a tabletop of the table 90b. For example, the tabletop of the table 90b is formed of a transparent material such as a glass plate or a transparent plastic plate. Then, the information projected by the output unit 124b is displayed on the top surface (display surface 20b) of the table 90b (through the tabletop). The method of displaying information on the display surface 20b by projecting the information from below the table 90b to the output unit 124b in this way is also referred to as “rear projection type”.

[0084] In addition, in the example illustrated in FIG. 2, the input unit 120b is provided on the display surface 20b. For example, the input unit 120b is constituted by a touch panel. In this case, the touch of the operation tool to the display surface 20b is detected by the touch panel, so that the operation input of the user can be received. However, the present invention is not limited to such an example, and the input unit 120b may be arranged below the table 90b while being separated from the table 90b, similarly to the information processing system 10a illustrated in FIG. 2. In this case, the input unit 120b can include a camera, and the camera can photograph the operation tool positioned on the display surface 20b over the tabletop of the table 90b. Then, the position of the operation tool can be detected based on the photographed image.

……
……
……