Qualcomm Patent | Representing Occlusion When Rendering For Computer-Mediated Reality Systems
Patent: Representing Occlusion When Rendering For Computer-Mediated Reality Systems
Publication Number: 20200107147
Publication Date: 20200402
Applicants: Qualcomm
Abstract
In general, techniques are described for modeling occlusions when rendering audio data. A device comprising a memory and one or more processors may perform the techniques. The memory may store audio data representative of a soundfield. The one or more processors may obtain occlusion metadata representative of an occlusion within the soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces. The one or more processors may obtain a location of the device, and obtain, based on the occlusion metadata and the location, a renderer by which to render the audio data into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces in which the device resides. The one or more processors may apply the renderer to the audio data to generate the speaker feeds.
[0001] This application claims the benefit of U.S. Provisional Ser. No. 62/740,085, entitled “REPRESENTING OCCULSION WHEN RENDERING FO COMPUTER-MEDIATED REALITY SYSTEMS,” filed Oct. 2, 2018, the entire contents of which are hereby incorporated by reference as if set forth in their entirety.
TECHNICAL FIELD
[0002] This disclosure relates to processing of media data, such as audio data.
BACKGROUND
[0003] Computer-mediated reality systems are being developed to allow computing devices to augment or add to, remove or subtract from, or generally modify existing reality experienced by a user. Computer-mediated reality systems may include, as a couple of examples, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of computer-mediated reality systems are generally related to the ability of such computer-mediated reality systems to provide a realistically immersive experience in terms of both the video and audio experience where the video and audio experience align in ways expected by the user. Although the human visual system is more sensitive than the human auditory systems (e.g., in terms of perceived localization of various objects within the scene), ensuring a adequate auditory experience is an increasingly import factor in ensuring a realistically immersive experience, particularly as the video experience improves to permit better localization of video objects that enable the user to better identify sources of audio content.
SUMMARY
[0004] This disclosure relates generally to auditory aspects of the user experience of computer-mediated reality systems, including virtual reality (VR), mixed reality (MR), augmented reality (AR), and/or any other type of extended reality (XR), and in addition to computer vision, and graphics systems. The techniques may enable modeling of occlusions when rendering audio data for the computer-mediated reality systems. Rather than only account for reflections in a given virtual environment, the techniques may enable the computer-mediated reality systems to address occlusions that may prevent audio waves (which may also be referred to a “sound”) represented by the audio data from propagating by various degrees throughout the virtual space. Furthermore, the techniques may enable different models based on different virtual environments, where for example a binaural room impulse response (BRIR) model may be used in virtual indoor environments, while a head related transfer function (HRTF) may be used in virtual outdoor environments.
[0005] In one example, the techniques are directed to a device comprising: a memory configured to store audio data representative of a soundfield; and one or more processors coupled to the memory, and configured to: obtain occlusion metadata representative of an occlusion within the soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; obtain a location of the device within the soundfield relative to the occlusion; obtain, based on the occlusion metadata and the location, a renderer by which to render the audio data into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces in which the device resides; and apply the renderer to the audio data to generate the speaker feeds.
[0006] In another example, the techniques are directed to a method comprising: obtaining, by a device, occlusion metadata representative of an occlusion within a soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; obtaining, by the device, a location of the device within the soundfield relative to the occlusion; obtaining, by the device, based on the occlusion metadata and the location, a renderer by which to render audio data representative of the soundfield into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces in which the device resides; and applying, by the device, the renderer to the audio data to generate the speaker feeds.
[0007] In another example, the techniques are directed to a device comprising: means for obtaining occlusion metadata representative of an occlusion within a soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; means for obtaining a location of the device within the soundfield relative to the occlusion; means for obtaining, based on the occlusion metadata and the location, a renderer by which to render audio data representative of the soundfield into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces in which the device resides; and means for applying the renderer to the audio data to generate the speaker feeds.
[0008] In another example, the techniques are directed to a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors of a device to: obtain, occlusion metadata representative of an occlusion within a soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; obtain a location of the device within the soundfield relative to the occlusion; obtain, based on the occlusion metadata and the location, a renderer by which to render audio data representative of the soundfield into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces in which the device resides; and apply the renderer to the audio data to generate the speaker feeds.
[0009] In another example, the techniques are directed to a device comprising: a memory configured to store audio data representative of a soundfield; and one or more processors coupled to the memory, and configured to: obtain occlusion metadata representative of an occlusion within the soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; specify, in a bitstream representative of the audio data, the occlusion metadata to enable a renderer to be obtained by which to render the audio data into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces.
[0010] In another example, the techniques are directed to a method comprising: obtaining, by a device, occlusion metadata representative of an occlusion within a soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; specifying, by the device, in a bitstream representative of audio data descriptive of the soundfield, the occlusion metadata to enable a renderer to be obtained by which to render the audio data into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces.
[0011] In another example, the techniques are directed to a device comprising: means for obtaining occlusion metadata representative of an occlusion within a soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; means for specifying, in a bitstream representative of audio data descriptive of the soundfield, the occlusion metadata to enable a renderer to be obtained by which to render the audio data into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces.
[0012] In another example, the techniques are directed to a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors of a device to: obtain occlusion metadata representative of an occlusion within a soundfield in terms of propagation of sound through the occlusion, the occlusion separating the soundfield into two or more sound spaces; and specify, in a bitstream representative of audio data descriptive of the soundfield, the occlusion metadata to enable a renderer to be obtained by which to render the audio data into one or more speaker feeds that account for propagation of the sound in one of the two or more sound spaces.
[0013] The details of one or more examples of this disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the techniques will be apparent from the description and drawings, and from the claims.
BRIEF DESCRIPTION OF DRAWINGS
[0014] FIGS. 1A and 1B are diagrams illustrating systems that may perform various aspects of the techniques described in this disclosure.
[0015] FIG. 2 is a block diagram illustrating an example of how the audio decoding device of FIG. 1A may apply various aspects of the techniques to facilitate occlusion aware rendering of audio data.
[0016] FIG. 3 is a block diagram illustrating another example how the audio decoding device of FIG. 1A may apply various aspects of the techniques to facilitate occlusion aware rendering of audio data.
[0017] FIG. 4 is a block diagram illustrating an example occlusion and the accompanying occlusion metadata that may be provided in accordance with various aspects of the techniques described in this disclosure.
[0018] FIG. 5 is a block diagram illustrating an example of an occlusion aware renderer that the audio decoding device of FIG. 1A may configure based on the occlusion metadata.
[0019] FIG. 6 is a block diagram illustrating how the audio decoding device of FIG. 1A may obtain, in accordance with various aspects of the techniques described in this disclosure, a renderer when an occlusion separates the soundfield into two sound spaces.
[0020] FIG. 7 is a block diagram illustrating an example portion of the audio bitstream of FIG. 1A formed in accordance with various aspects of the techniques described in this disclosure.
[0021] FIG. 8 is a block diagram of the inputs used to configure the occlusion aware renderer of FIG. 1 in accordance with various aspects of the techniques described in this disclosure.
[0022] FIGS. 9A and 9B are diagrams illustrating example systems that may perform various aspects of the techniques described in this disclosure.
[0023] FIGS. 10A and 10B are diagrams illustrating other example systems that may perform various aspects of the techniques described in this disclosure.
[0024] FIG. 11 is a flowchart illustrating example operation of the systems of FIGS. 1A and 1B in performing various aspects of the techniques described in this disclosure.
[0025] FIG. 12 is a flowchart illustrating example operation of the audio playback system shown in the example of FIG. 1A in performing various aspects of the techniques described in this disclosure.
[0026] FIG. 13 is a block diagram of the audio playback device shown in the examples of FIGS. 1A and 1B in performing various aspects of the techniques described in this disclosure.
[0027] FIG. 14 illustrates an example of a wireless communications system that supports audio streaming in accordance with aspects of the present disclosure.
DETAILED DESCRIPTION
[0028] There are a number of different ways to represent a soundfield. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to the 5.1 surround sound format, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based format that localizes audio channels to particular locations around the listener in order to recreate a soundfield.
[0029] Object-based audio formats may refer to formats in which audio objects, often encoded using pulse-code modulation (PCM) and referred to as PCM audio objects, are specified in order to represent the soundfield. Such audio objects may include metadata identifying a location of the audio object relative to a listener or other point of reference in the soundfield, such that the audio object may be rendered to one or more speaker channels for playback in an effort to recreate the soundfield. The techniques described in this disclosure may apply to any of the foregoing formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.
[0030] Scene-based audio formats may include a hierarchical set of elements that define the soundfield in three dimensions. One example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The following expression demonstrates a description or representation of a soundfield using SHC:
p i ( t , r r , .theta. r , .PHI. r ) = .omega. = 0 .infin. [ 4 .pi. .omega. = 0 .infin. j n ( kr r ) m = - n n A n m ( k ) Y n m ( .theta. r , .PHI. r ) ] e j .omega. t , ##EQU00001##
[0031] The expression shows that the pressure p.sub.i at any point {r.sub.r, B.sub.r, .phi..sub.r} of the soundfield, at time t, can be represented uniquely by the SHC, A.sub.n.sup.m(k).* Here*
k = .omega. c , ##EQU00002##
c is the speed of sound (.about.343 m/s), {r.sub.r, .theta..sub.r, .phi..sub.r} is a point of reference (or observation point), j.sub.n() is the spherical Bessel function of order n, and Y.sub.n.sup.m(.theta..sub.r, .phi..sub.r) are the spherical harmonic basis functions (which may also be referred to as a spherical basis function) of order n and suborder m. It can be recognized that the term in square brackets is a frequency-domain representation of the signal (i.e., S(.omega., r.sub.r, B.sub.r, .phi..sub.r)) which can be approximated by various time-frequency transformations, such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or a wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.
[0032] The SHC A.sub.n.sup.m (k) can either be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, they can be derived from channel-based or object-based descriptions of the soundfield. The SHC (which also may be referred to as ambisonic coefficients) represent scene-based audio, where the SHC may be input to an audio encoder to obtain encoded SHC that may promote more efficient transmission or storage. For example, a fourth-order representation involving (1+4).sup.2 (25, and hence fourth order) coefficients may be used.
[0033] As noted above, the SHC may be derived from a microphone recording using a microphone array. Various examples of how SHC may be physically acquired from microphone arrays are described in Poletti, M., “Three-Dimensional Surround Sound Systems Based on Spherical Harmonics,” J. Audio Eng. Soc., Vol. 53, No. 11, 2005 November, pp. 1004-1025.
[0034] The following equation may illustrate how the SHCs may be derived from an object-based description. The coefficients A.sub.n.sup.m (k) for the soundfield corresponding to an individual audio object may be expressed as:
A.sub.n.sup.m (k)=g (.omega.)(-4.pi.ik)h.sub.n.sup.(2)(kr.sub.s)Y.sub.n.sup.m*(.theta..sub.s,- .phi..sub.s),
where i is {square root over (-1)}, h.sub.n.sup.(2)() is the spherical Hankel function (of the second kind) of order n, and {r.sub.s, .theta..sub.s, .phi..sub.s} is the location of the object. Knowing the object source energy g(.omega.) as a function of frequency (e.g., using time-frequency analysis techniques, such as performing a fast Fourier transform on the pulse code modulated–PCM–stream) may enable conversion of each PCM object and the corresponding location into the SHC A.sub.n.sup.m(k) Further, it can be shown (since the above is a linear and orthogonal decomposition) that the A.sub.n.sup.m (k) coefficients for each object are additive. In this manner, a number of PCM objects can be represented by the A.sub.n.sup.m (k) coefficients (e.g., as a sum of the coefficient vectors for the individual objects). The coefficients may contain information about the soundfield (the pressure as a function of 3D coordinates), and the above represents the transformation from individual objects to a representation of the overall soundfield, in the vicinity of the observation point {r.sub.r, .theta..sub.r, .phi..sub.r}.
[0035] Computer-mediated reality systems (which may also be referred to as “extended reality systems,” or “XR systems”) are being developed to take advantage of many of the potential benefits provided by ambisonic coefficients. For example, ambisonic coefficients may represent a soundfield in three dimensions in a manner that potentially enables accurate three-dimensional (3D) localization of sound sources within the soundfield. As such, XR devices may render the ambisonic coefficients to speaker feeds that, when played via one or more speakers, accurately reproduce the soundfield.
[0036] The use of ambisonic coefficients for XR may enable development of a number of use cases that rely on the more immersive soundfields provided by the ambisonic coefficients, particularly for computer gaming applications and live video streaming applications. In these highly dynamic use cases that rely on low latency reproduction of the soundfield, the XR devices may prefer ambisonic coefficients over other representations that are more difficult to manipulate or involve complex rendering. More information regarding these use cases is provided below with respect to FIGS. 1A and 1B.
[0037] While described in this disclosure with respect to the VR device, various aspects of the techniques may be performed in the context of other devices, such as a mobile device. In this instance, the mobile device (such as a so-called smartphone) may present the displayed world via a screen, which may be mounted to the head of the user 102 or viewed as would be done when normally using the mobile device. As such, any information on the screen can be part of the mobile device. The mobile device may be able to provide tracking information 41 and thereby allow for both a VR experience (when head mounted) and a normal experience to view the displayed world, where the normal experience may still allow the user to view the displayed world proving a VR-lite-type experience (e.g., holding up the device and rotating or translating the device to view different portions of the displayed world).
[0038] FIGS. 1A and 1B are diagrams illustrating systems that may perform various aspects of the techniques described in this disclosure. As shown in the example of FIG. 1A, system 10 includes a source device 12 and a content consumer device 14. While described in the context of the source device 12 and the content consumer device 14, the techniques may be implemented in any context in which any hierarchical representation of a soundfield is encoded to form a bitstream representative of the audio data. Moreover, the source device 12 may represent any form of computing device capable of generating hierarchical representation of a soundfield, and is generally described herein in the context of being a VR content creator device. Likewise, the content consumer device 14 may represent any form of computing device capable of implementing the audio stream interpolation techniques described in this disclosure as well as audio playback, and is generally described herein in the context of being a VR client device.
[0039] The source device 12 may be operated by an entertainment company or other entity that may generate multi-channel audio content for consumption by operators of content consumer devices, such as the content consumer device 14. In many VR scenarios, the source device 12 generates audio content in conjunction with video content. The source device 12 includes a content capture device 300 and a content soundfield representation generator 302.
[0040] The content capture device 300 may be configured to interface or otherwise communicate with one or more microphones 5A-5N (“microphones 5”). The microphones 5 may represent an Eigenmike.RTM. or other type of 3D audio microphone capable of capturing and representing the soundfield as corresponding scene-based audio data 11A-11N (which may also be referred to as ambisonic coefficients 11A-11N or “ambisonic coefficients 11”). In the context of scene-based audio data 11 (which is another way to refer to the ambisonic coefficients 11”), each of the microphones 5 may represent a cluster of microphones arranged within a single housing according to set geometries that facilitate generation of the ambisonic coefficients 11. As such, the term microphone may refer to a cluster of microphones (which are actually geometrically arranged transducers) or a single microphone (which may be referred to as a spot microphone).
[0041] The ambisonic coefficients 11 may represent one example of an audio stream. As such, the ambisonic coefficients 11 may also be referred to as audio streams 11. Although described primarily with respect to the ambisonic coefficients 11, the techniques may be performed with respect to other types of audio streams, including pulse code modulated (PCM) audio streams, channel-based audio streams, object-based audio streams, etc.
[0042] The content capture device 300 may, in some examples, include an integrated microphone that is integrated into the housing of the content capture device 300. The content capture device 300 may interface wirelessly or via a wired connection with the microphones 5. Rather than capture, or in conjunction with capturing, audio data via the microphones 5, the content capture device 300 may process the ambisonic coefficients 11 after the ambisonic coefficients 11 are input via some type of removable storage, wirelessly, and/or via wired input processes, or alternatively or in conjunction with the foregoing, generated or otherwise created (from stored sound samples, such as is common in gaming applications, etc.). As such, various combinations of the content capture device 300 and the microphones 5 are possible.
[0043] The content capture device 300 may also be configured to interface or otherwise communicate with the soundfield representation generator 302. The soundfield representation generator 302 may include any type of hardware device capable of interfacing with the content capture device 300. The soundfield representation generator 302 may use the ambisonic coefficients 11 provided by the content capture device 300 to generate various representations of the same soundfield represented by the ambisonic coefficients 11.
[0044] For instance, to generate the different representations of the soundfield using ambisonic coefficients (which again is one example of the audio streams), the soundfield representation generator 24 may use a coding scheme for ambisonic representations of a soundfield, referred to as Mixed Order Ambisonics (MOA) as discussed in more detail in U.S. application Ser. No. 15/672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FO COMPUTER-MEDIATED REALITY SYSTEMS,” filed Aug. 8, 2017, and published as U.S. patent publication no. 20190007781 on Jan. 3, 2019.
[0045] To generate a particular MOA representation of the soundfield, the soundfield representation generator 24 may generate a partial subset of the full set of ambisonic coefficients. For instance, each MOA representation generated by the soundfield representation generator 24 may provide precision with respect to some areas of the soundfield, but less precision in other areas. In one example, an MOA representation of the soundfield may include eight (8) uncompressed ambisonic coefficients, while the third order ambisonic representation of the same soundfield may include sixteen (16) uncompressed ambisonic coefficients. As such, each MOA representation of the soundfield that is generated as a partial subset of the ambisonic coefficients may be less storage-intensive and less bandwidth intensive (if and when transmitted as part of the bitstream 27 over the illustrated transmission channel) than the corresponding third order ambisonic representation of the same soundfield generated from the ambisonic coefficients.
[0046] Although described with respect to MOA representations, the techniques of this disclosure may also be performed with respect to first-order ambisonic (FOA) representations in which all of the ambisonic coefficients associated with a first order spherical basis function and a zero order spherical basis function are used to represent the soundfield. In other words, rather than represent the soundfield using a partial, non-zero subset of the ambisonic coefficients, the soundfield representation generator 302 may represent the soundfield using all of the ambisonic coefficients for a given order N, resulting in a total of ambisonic coefficients equaling (N+1).sup.2.
[0047] In this respect, the ambisonic audio data (which is another way to refer to the ambisonic coefficients in either MOA representations or full order representations, such as the first-order representation noted above) may include ambisonic coefficients associated with spherical basis functions having an order of one or less (which may be referred to as “1.sup.st order ambisonic audio data”), ambisonic coefficients associated with spherical basis functions having a mixed order and suborder (which may be referred to as the “MOA representation” discussed above), or ambisonic coefficients associated with spherical basis functions having an order greater than one (which is referred to above as the “full order representation”).
[0048] The content capture device 300 may, in some examples, be configured to wirelessly communicate with the soundfield representation generator 302. In some examples, the content capture device 300 may communicate, via one or both of a wireless connection or a wired connection, with the soundfield representation generator 302. Via the connection between the content capture device 300 and the soundfield representation generator 302, the content capture device 300 may provide content in various forms of content, which, for purposes of discussion, are described herein as being portions of the ambisonic coefficients 11.
[0049] In some examples, the content capture device 300 may leverage various aspects of the soundfield representation generator 302 (in terms of hardware or software capabilities of the soundfield representation generator 302). For example, the soundfield representation generator 302 may include dedicated hardware configured to (or specialized software that when executed causes one or more processors to) perform psychoacoustic audio encoding (such as a unified speech and audio coder denoted as “USAC” set forth by the Moving Picture Experts Group (MPEG), the MPEG-H 3D audio coding standard, the MPEG-I Immersive Audio standard, or proprietary standards, such as AptX.TM. (including various versions of AptX such as enhanced AptX-E-AptX, AptX live, AptX stereo, and AptX high definition–AptX-HD), advanced audio coding (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey’s Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Windows Media Audio (WMA).
[0050] The content capture device 300 may not include the psychoacoustic audio encoder dedicated hardware or specialized software and instead provide audio aspects of the content 301 in a non-psychoacoustic audio coded form. The soundfield representation generator 302 may assist in the capture of content 301 by, at least in part, performing psychoacoustic audio encoding with respect to the audio aspects of the content 301.
[0051] The soundfield representation generator 302 may also assist in content capture and transmission by generating one or more bitstreams 21 based, at least in part, on the audio content (e.g., MOA representations, third order ambisonic representations, and/or first order ambisonic representations) generated from the ambisonic coefficients 11. The bitstream 21 may represent a compressed version of the ambisonic coefficients 11 (and/or the partial subsets thereof used to form MOA representations of the soundfield) and any other different types of the content 301 (such as a compressed version of spherical video data, image data, or text data).
[0052] The soundfield representation generator 302 may generate the bitstream 21 for transmission, as one example, across a transmission channel, which may be a wired or wireless channel, a data storage device, or the like. The bitstream 21 may represent an encoded version of the ambisonic coefficients 11 (and/or the partial subsets thereof used to form MOA representations of the soundfield) and may include a primary bitstream and another side bitstream, which may be referred to as side channel information. In some instances, the bitstream 21 representing the compressed version of the ambisonic coefficients 11 may conform to bitstreams produced in accordance with the MPEG-H 3D audio coding standard.
[0053] The content consumer device 14 may be operated by an individual, and may represent a VR client device. Although described with respect to a VR client device, content consumer device 14 may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device (or any other type of head-mounted display device or extended reality–XR– device), a standard computer, a headset, headphones, or any other device capable of tracking head movements and/or general translational movements of the individual operating the client consumer device 14. As shown in the example of FIG. 1A, the content consumer device 14 includes an audio playback system 16A, which may refer to any form of audio playback system capable of rendering ambisonic coefficients (whether in form of first order, second order, and/or third order ambisonic representations and/or MOA representations) for playback as multi-channel audio content.
[0054] The content consumer device 14 may retrieve the bitstream 21 directly from the source device 12. In some examples, the content consumer device 12 may interface with a network, including a fifth generation (5G) cellular network, to retrieve the bitstream 21 or otherwise cause the source device 12 to transmit the bitstream 21 to the content consumer device 14.
[0055] While shown in FIG. 1A as being directly transmitted to the content consumer device 14, the source device 12 may output the bitstream 21 to an intermediate device positioned between the source device 12 and the content consumer device 14. The intermediate device may store the bitstream 21 for later delivery to the content consumer device 14, which may request the bitstream. The intermediate device may comprise a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream 21 for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming the bitstream 21 (and possibly in conjunction with transmitting a corresponding video data bitstream) to subscribers, such as the content consumer device 14, requesting the bitstream 21.
[0056] Alternatively, the source device 12 may store the bitstream 21 to a storage medium, such as a compact disc, a digital video disc, a high definition video disc or other storage media, most of which are capable of being read by a computer and therefore may be referred to as computer-readable storage media or non-transitory computer-readable storage media. In this context, the transmission channel may refer to the channels by which content stored to the mediums are transmitted (and may include retail stores and other store-based delivery mechanism). In any event, the techniques of this disclosure should not therefore be limited in this respect to the example of FIG. 1A.
[0057] As noted above, the content consumer device 14 includes the audio playback system 16. The audio playback system 16 may represent any system capable of playing back multi-channel audio data. The audio playback system 16A may include a number of different audio renderers 22. The renderers 22 may each provide for a different form of audio rendering, where the different forms of rendering may include one or more of the various ways of performing vector-base amplitude panning (VBAP), and/or one or more of the various ways of performing soundfield synthesis. As used herein, “A and/or B” means “A or B”, or both “A and B”.
[0058] The audio playback system 16A may further include an audio decoding device 24. The audio decoding device 24 may represent a device configured to decode bitstream 21 to output reconstructed ambisonic coefficients 11A’-11N’ (which may form the full first, second, and/or third order ambisonic representation or a subset thereof that forms an MOA representation of the same soundfield or decompositions thereof, such as the predominant audio signal, ambient ambisonic coefficients, and the vector based signal described in the MPEG-H 3D Audio Coding Standard and/or the MPEG-I Immersive Audio standard).
[0059] As such, the ambisonic coefficients 11A’-11N’ (“ambisonic coefficients 11’”) may be similar to a full set or a partial subset of the ambisonic coefficients 11, but may differ due to lossy operations (e.g., quantization) and/or transmission via the transmission channel. The audio playback system 16 may, after decoding the bitstream 21 to obtain the ambisonic coefficients 11’, obtain ambisonic audio data 15 from the different streams of ambisonic coefficients 11’, and render the ambisonic audio data 15 to output speaker feeds 25. The speaker feeds 25 may drive one or more speakers (which are not shown in the example of FIG. 1A for ease of illustration purposes). Ambisonic representations of a soundfield may be normalized in a number of ways, including N3D, SN3D, FuMa, N2D, or SN2D.
[0060] To select the appropriate renderer or, in some instances, generate an appropriate renderer, the audio playback system 16A may obtain loudspeaker information 13 indicative of a number of loudspeakers and/or a spatial geometry of the loudspeakers. In some instances, the audio playback system 16A may obtain the loudspeaker information 13 using a reference microphone and outputting a signal to activate (or, in other words, drive) the loudspeakers in such a manner as to dynamically determine, via the reference microphone, the loudspeaker information 13. In other instances, or in conjunction with the dynamic determination of the loudspeaker information 13, the audio playback system 16A may prompt a user to interface with the audio playback system 16A and input the loudspeaker information 13.
[0061] The audio playback system 16A may select one of the audio renderers 22 based on the loudspeaker information 13. In some instances, the audio playback system 16A may, when none of the audio renderers 22 are within some threshold similarity measure (in terms of the loudspeaker geometry) to the loudspeaker geometry specified in the loudspeaker information 13, generate the one of audio renderers 22 based on the loudspeaker information 13. The audio playback system 16A may, in some instances, generate one of the audio renderers 22 based on the loudspeaker information 13 without first attempting to select an existing one of the audio renderers 22.
[0062] When outputting the speaker feeds 25 to headphones, the audio playback system 16A may utilize one of the renderers 22 that provides for binaural rendering using head-related transfer functions (HRTF) or other functions capable of rendering to left and right speaker feeds 25 for headphone speaker playback. The terms “speakers” or “transducer” may generally refer to any speaker, including loudspeakers, headphone speakers, etc. One or more speakers may then playback the rendered speaker feeds 25.
[0063] Although described as rendering the speaker feeds 25 from the ambisonic audio data 15, reference to rendering of the speaker feeds 25 may refer to other types of rendering, such as rendering incorporated directly into the decoding of the ambisonic audio data 15 from the bitstream 21. An example of the alternative rendering can be found in Annex G of the MPEG-H 3D audio coding standard, where rendering occurs during the predominant signal formulation and the background signal formation prior to composition of the soundfield. As such, reference to rendering of the ambisonic audio data 15 should be understood to refer to both rendering of the actual ambisonic audio data 15 or decompositions or representations thereof of the ambisonic audio data 15 (such as the above noted predominant audio signal, the ambient ambisonic coefficients, and/or the vector-based signal–which may also be referred to as a V-vector).
[0064] As described above, the content consumer device 14 may represent a VR device in which a human wearable display is mounted in front of the eyes of the user operating the VR device. FIGS. 9A and 9B are diagrams illustrating examples of VR devices 400A and 400B. In the example of FIG. 9A, the VR device 400A is coupled to, or otherwise includes, headphones 404, which may reproduce a soundfield represented by the ambisonic audio data 15 (which is another way to refer to ambisonic coefficients 15) through playback of the speaker feeds 25. The speaker feeds 25 may represent an analog or digital signal capable of causing a membrane within the transducers of headphones 404 to vibrate at various frequencies. Such a process is commonly referred to as driving the headphones 404.
[0065] Video, audio, and other sensory data may play important roles in the VR experience. To participate in a VR experience, a user 402 may wear the VR device 400A (which may also be referred to as a VR headset 400A) or other wearable electronic device. The VR client device (such as the VR headset 400A) may track head movement of the user 402, and adapt the video data shown via the VR headset 400A to account for the head movements, providing an immersive experience in which the user 402 may experience a virtual world shown in the video data in visual three dimensions.
[0066] While VR (and other forms of AR and/or MR, which may generally be referred to as a computer mediated reality device) may allow the user 402 to reside in the virtual world visually, often the VR headset 400A may lack the capability to place the user in the virtual world audibly. In other words, the VR system (which may include a computer responsible for rendering the video data and audio data–that is not shown in the example of FIG. 9A for ease of illustration purposes, and the VR headset 400A) may be unable to support full three dimension immersion audibly.
[0067] FIG. 9B is a diagram illustrating an example of a wearable device 400B that may operate in accordance with various aspect of the techniques described in this disclosure. In various examples, the wearable device 400B may represent a VR headset (such as the VR headset 400A described above), an AR headset, an MR headset, or any other type of XR headset. Augmented Reality “AR” may refer to computer rendered image or data that is overlaid over the real world where the user is actually located. Mixed Reality “MR” may refer to computer rendered image or data that is world locked to a particular location in the real world, or may refer to a variant on VR in which part computer rendered 3D elements and part photographed real elements are combined into an immersive experience that simulates the user’s physical presence in the environment. Extended Reality “XR” may represent a catchall term for VR, AR, and MR. More information regarding terminology for XR can be found in a document by Jason Peterson, entitled “Virtual Reality, Augmented Reality, and Mixed Reality Definitions,” and dated Jul. 7, 2017.
[0068] The wearable device 400B may represent other types of devices, such as a watch (including so-called “smart watches”), glasses (including so-called “smart glasses”), headphones (including so-called “wireless headphones” and “smart headphones”), smart clothing, smart jewelry, and the like. Whether representative of a VR device, a watch, glasses, and/or headphones, the wearable device 400B may communicate with the computing device supporting the wearable device 400B via a wired connection or a wireless connection.
[0069] In some instances, the computing device supporting the wearable device 400B may be integrated within the wearable device 400B and as such, the wearable device 400B may be considered as the same device as the computing device supporting the wearable device 400B. In other instances, the wearable device 400B may communicate with a separate computing device that may support the wearable device 400B. In this respect, the term “supporting” should not be understood to require a separate dedicated device but that one or more processors configured to perform various aspects of the techniques described in this disclosure may be integrated within the wearable device 400B or integrated within a computing device separate from the wearable device 400B.
[0070] For example, when the wearable device 400B represents an example of the VR device 400B, a separate dedicated computing device (such as a personal computer including the one or more processors) may render the audio and visual content, while the wearable device 400B may determine the translational head movement upon which the dedicated computing device may render, based on the translational head movement, the audio content (as the speaker feeds) in accordance with various aspects of the techniques described in this disclosure. As another example, when the wearable device 400B represents smart glasses, the wearable device 400B may include the one or more processors that both determine the translational head movement (by interfacing within one or more sensors of the wearable device 400B) and render, based on the determined translational head movement, the speaker feeds.
[0071] As shown, the wearable device 400B includes one or more directional speakers, and one or more tracking and/or recording cameras. In addition, the wearable device 400B includes one or more inertial, haptic, and/or health sensors, one or more eye-tracking cameras, one or more high sensitivity audio microphones, and optics/projection hardware. The optics/projection hardware of the wearable device 400B may include durable semi-transparent display technology and hardware.
[0072] The wearable device 400B also includes connectivity hardware, which may represent one or more network interfaces that support multimode connectivity, such as 4G communications, 5G communications, Bluetooth, etc. The wearable device 400B also includes one or more ambient light sensors, and bone conduction transducers. In some instances, the wearable device 400B may also include one or more passive and/or active cameras with fisheye lenses and/or telephoto lenses. Although not shown in FIG. 5B, the wearable device 400B also may include one or more light emitting diode (LED) lights. In some examples, the LED light(s) may be referred to as “ultra bright” LED light(s). The wearable device 400B also may include one or more rear cameras in some implementations. It will be appreciated that the wearable device 400B may exhibit a variety of different form factors.
[0073] Furthermore, the tracking and recording cameras and other sensors may facilitate the determination of translational distance. Although not shown in the example of FIG. 9B, wearable device 400B may include other types of sensors for detecting translational distance.
[0074] Although described with respect to particular examples of wearable devices, such as the VR device 400B discussed above with respect to the examples of FIG. 9B and other devices set forth in the examples of FIGS. 1A and 1B, a person of ordinary skill in the art would appreciate that descriptions related to FIGS. 1A-1B may apply to other examples of wearable devices. For example, other wearable devices, such as smart glasses, may include sensors by which to obtain translational head movements. As another example, other wearable devices, such as a smart watch, may include sensors by which to obtain translational movements. As such, the techniques described in this disclosure should not be limited to a particular type of wearable device, but any wearable device may be configured to perform the techniques described in this disclosure.
[0075] In any event, the audio aspects of VR have been classified into three separate categories of immersion. The first category provides the lowest level of immersion, and is referred to as three degrees of freedom (3DOF). 3DOF refers to audio rendering that accounts for movement of the head in the three degrees of freedom (yaw, pitch, and roll), thereby allowing the user to freely look around in any direction. 3DOF, however, cannot account for translational head movements in which the head is not centered on the optical and acoustical center of the soundfield.
[0076] The second category, referred to 3DOF plus (3DOF+), provides for the three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational movements due to the head movements away from the optical center and acoustical center within the soundfield. 3DOF+ may provide support for perceptual effects such as motion parallax, which may strengthen the sense of immersion.
[0077] The third category, referred to as six degrees of freedom (6DOF), renders audio data in a manner that accounts for the three degrees of freedom in term of head movements (yaw, pitch, and roll) but also accounts for translation of the user in space (x, y, and z translations). The spatial translations may be induced by sensors tracking the location of the user in the physical world or by way of an input controller.
[0078] 3DOF rendering is the current state of the art for audio aspects of VR. As such, the audio aspects of VR are less immersive than the video aspects, thereby potentially reducing the overall immersion experienced by the user, and introducing localization errors (e.g., such as when the auditory playback does not match or correlate exactly to the visual scene).
[0079] Furthermore, how sound is modeled in relation to the virtual environment is still being developed to enable more realistic sound propagation when various environmental objects may impact propagation of sound within the virtual environment. As such, audio immersion may be degraded when sounds appear to propagate through the virtual environment in ways that do not accurately reflect when the user of the VR headset 400 expects when confronted with real environments having similar geometries and objects. As one example, a common VR audio software developers kit may only permit for modeling of direct reflections of sounds off of objects (which may also be referred to as “occlusions”), such as walls, doors (where the occlusion metadata 305 for a door and other movable physical–virtually–occlusions may change as a result of the door being in different states of openness or closedness), etc. that separate the soundfield into two or more sound spaces, and do not account for how sound may propagate through such objects, reducing audio immersion who expects loud sounds (such as a gunshot, a scream, a helicopter, etc.) to propagate through some objects like walls and doors.
[0080] In accordance with the techniques described in this disclosure, the source device 12 may obtain occlusion metadata (which may represent a portion of the metadata 305, and as such may be referred to as “occlusion metadata 305”) representative of an occlusion within the soundfield (represented by the edited audio data, which may form a portion of edited content 303 and as such may be denoted “edited audio data 305”) in terms of propagation of sound through the occlusion. An audio editor may, when editing audio data 301 and in some examples, specify the occlusion metadata 305.
[0081] Alternatively or in combination with manual entry of occlusion metadata 305, the content editing device may automatically generate the occlusion metadata 305 (e.g., via software that, when executed, configures the content editor device 304 to automatically generate the occlusion metadata 305). In some instances, the audio editor may identify the occlusions and the content editor device 304 may automatically associate pre-defined occlusion metadata 305 with the manually identified occlusion. In any event, the content editor device 304 may obtain the occlusion metadata 305 and provide the occlusion metadata 305 to the soundfield representation generator 302.
[0082] The soundfield representation generator 302 may represent one example of a device or other unit configured to specify, in the audio bitstream 21 representative of the edited audio content 303 (which may refer to one of the one or more bitstreams 21), the occlusion metadata 305 to enable a renderer 22 to be obtained (by, e.g., the audio playback system 16) by which to render the edited audio content 303 into one or more speaker feeds 25 to model (or in other words, take into account of) how the sound propagates in one of two or more sound spaces separated by the occlusion (or, in slightly different words, that account for the propagation of sound in one of the two or more sound spaces separated by the occlusion).
[0083] The audio decoding device 24 may obtain, in some examples from the audio bitstream 21, the occlusion metadata 305 representative of the occlusion within the soundfield in terms of propagation of sound through the occlusion, where again the occlusion may separate the soundfield into two or more sound spaces. The audio decoding device 24 may also obtain a location 17 of the device (which in this instance may refer to the audio playback system 16 of which one example is the VR device) within the soundfield relative to the occlusion.
[0084] That is, the audio playback system 16 may interface with a tracking device 306, which represents a device configured to obtain the location 17 of the device. The audio playback system 16 may translate the physical location 17 within an actual space into a location within the virtual environment, and identify a location 317 of the audio playback system 16 relative to the location of the occlusion. The audio playback system 16 may obtain, based on the occlusion metadata 305 and the location 317, an occlusion-aware renderer of the renderers 22 by which to render the audio data 15 into one or more speaker feeds to model how the sound propagates in one of the two or more sound spaces in which the audio playback system 16 resides. The audio playback system 16 may then apply the occlusion-aware renderer (which may be denoted as “occlusion-aware renderer 22”) to generate the speaker feeds 25.
[0085] The occlusion metadata 305 may include any combination of a number of different types of metadata, including one or more of a volume attenuation factor, a direct path only indication, a low pass filter description, and an indication of the location of the occlusion. The volume attenuation factor may be representative of an amount of volume associated with the audio data 15 is reduced while passing through the occlusion. The direct path only indication may be representative of whether a direct path exists for the audio data 15 or reverberation processing is to be applied (via the occlusion-aware renderer 22) to the audio data 15. The low pass filter description may be representative of coefficients to describe a low pass filter or a parametric description of the low pass filter (as integrated into or applied along with the occlusion-aware renderer 22).
[0086] The audio decoding device 24 may utilize the occlusion metadata 305 to generate the occlusion-aware renderer 22 that mixes live, prerecorded and synthetic audio content for 3DOF or 6DOF rendering. The occlusion metadata 305 may define information of occlusion acoustic characteristics that enables the audio decoding device 24 to identify how the sound spaces interact. In other words, the occlusion metadata 305 may define boundaries of the sound space, diffraction (or in other words shadowing) relative to the occlusion, absorption (or in other words leakage) relative to the occlusion, and an environment in which the occlusion is located.
[0087] The audio decoding device 24 may utilize the occlusion metadata 305 in any number of ways to generate the occlusion-aware renderer 22. For example, the audio decoding device 24 may utilize the occlusion metadata 305 as inputs to discrete mathematical equations. As another example, the audio decoding device 24 may utilize the occlusion metadata 305 as inputs to empirically derived filters. As yet another example, the audio decoding device 24 may utilize the occlusion metadata 305 as inputs to machine learning algorithms used to match the effects of the sound spaces. The audio decoding device 24 may also, in some examples, utilize any combination of the foregoing examples to generate the occlusion-aware renderer 22, including allowing for manual intervention to override the foregoing examples (such as for artistic purposes). An example of how various aspects of the techniques described in this disclosure may be applied to potentially improve rendering of audio data to account for occlusions and increase audio immersion is further described with respect to the example of FIG. 2.
[0088] Although described with respect to a VR device as shown in the example of FIG. 2, the techniques may be performed by other types of wearable devices, including watches (such as so-called “smart watches”), glasses (such as so-called “smart glasses”), headphones (including wireless headphones coupled via a wireless connection, or smart headphones coupled via wired or wireless connection), and any other type of wearable device. As such, the techniques may be performed by any type of wearable device by which a user may interact with the wearable device while worn by the user.
[0089] FIG. 2 is a block diagram illustrating an example of how the audio decoding device of FIG. 1A may apply various aspects of the techniques to facilitate occlusion aware rendering of audio data. In the example of FIG. 3, the audio decoding device 24 may obtain the audio data 15 representative of two soundfields 450A and 450B, which overlap at portion 452. When multiple soundfields 450A and 450B overlap, the audio decoding device 24 may obtain occlusion metadata 305 that identifies that the boundaries of the soundfields 450A and 450B overlap and to what extent one of the soundfields 450A and 450B may occlude the other one of the soundfields 450A and 450B.
[0090] More specifically, when the location 317 indicates that the audio playback system 16 is located at location 454A (denoted “L.sub.1”), the audio decoding device 24 may determine that part of the soundfield 450A is occluded by a part of the soundfield 450B, and generate the occlusion-aware renderer 22 to account for the occlusion. When the location 317 indicates that the audio playback system 16 is located at location 404B (denoted “L.sub.2”), the audio decoding device 24 may determine that part of the soundfield 450B is occluded by a part of the soundfield 450A, and generate the occlusion-aware renderer 22 to account for the occlusion.
[0091] In the example of FIG. 2, the overlap portion 452 of soundfields 450A and 450B includes two sound spaces 456A and 456B. The occlusion metadata 305 may include a sound space boundary for each of the two sound spaces 456A and 456B, which may enable the audio decoding device 24 to obtain the occlusion-aware renderer 22 that potentially reflects the extent of the occlusion due to the overlap of the two soundfields 450A and 450B. As such, the occlusion may also refer to overlapping soundfields 450A and 450B in addition to referring to virtual objects that may obstruct the propagation of sound. The occlusion may, as a result, refer to any physical interaction (which in the example of FIG. 2 refers to the interaction of sound waves) that impacts the propagation of sound.
[0092] The occlusion metadata 305 may also include how to transition occlusion-aware rendering when the user of the audio playback system 16 moves within the soundfields 450A and 450B. For example, the audio decoding device 24 may obtain, based on the occlusion metadata 305, the occlusion-aware renderer 22 that transitions background components of the audio data 15 to foreground components when the location 317 of the user of the audio playback system 16 moves toward the edge of the portion 452.
[0093] The occlusion metadata 305 may also include, as noted above, an indication of the occlusion such that the audio decoding device 24 may obtain a distance of the occlusion (e.g., the portion 452) relative to the location 317 of the audio playback system 16. When the soundfield is occluded from a significant distance (e.g., such as above some threshold distance), the audio decoding device 24 may generate the occlusion-aware renderer 22 to model the occlusion as a mono source, which is then rendered according the occlusion-aware renderer. As an example, assume that the location 317 indicates that the audio playback system 16 is located at location 454A and there is a barrier between locations 454A and 454B (denoted “L.sub.2”, the audio decoding device 24 may generate the occlusion-aware renderer 22 to model the soundfield 450B as an occluded point source. Further information regarding how occlusion-aware rendering is performed when two soundfields interact is described with respect to FIG. 3.
[0094] FIG. 3 is a block diagram illustrating another example how the audio decoding device of FIG. 1A may apply various aspects of the techniques to facilitate occlusion aware rendering of audio data. In the example of FIG. 3, the audio decoding device 24 may obtain the audio data 15 representative of two soundfields 460A and 460B defined by the audio data 15A-15E and 15F-15H. As further shown in the example of FIG. 3, soundfield 460A includes two regions 464A and 464B represented by the audio data 15A-15B and 15C-15E, and soundfield 460B includes a single region 464C represented by the audio data 15F-15H.
[0095] Assume a scenario in which the user is able to move from the soundfield 460A to the soundfield 460B (or vice versa from the soundfield 460B to the soundfield 460A). In this scenario, the audio decoding device 24 may obtain occlusion metadata 305 that indicates whether or not sounds from the soundfield 460A may be heard in (or, in other words, propagates to) the soundfield 460B (and vice versa from the soundfield 460B may be heard in the soundfield 460A). The occlusion metadata 305 may in this respect differentiate between two different soundfields 460A and 460B.
[0096] Further, the audio decoding device 24 may receive the audio data 15A-15G grouped by each of regions 464A-464C. The content editing device 304 may associate different portions of the occlusion metadata 305 with each of the regions 464A-464C (or, in other words, with multiple audio data–e.g., a first portion of the occlusion metadata 305 with the audio data 15A-15B, a second portion of the occlusion metadata 305 with 15C-15E, and a third portion of the occlusion metadata 305 with 15F-15G). The association of different portions of the occlusion metadata 305 with each of the regions 464A-464C may promote more efficient transmission of the occlusion metadata 305 as less occlusion metadata may be sent, promoting more compact bitstreams that reduce memory and bandwidth consumption and processing cycles when generating the audio bitstream 21.
[0097] In this way, the audio decoding device 24 may obtain, based on the occlusion metadata 305 and the location 317, a first renderer for different sets of audio data (such as a group of audio objects–e.g., audio objects 15A and 15B), and apply the first renderer to the first group of audio objects to obtain first speaker feeds. The audio decoding device 24 may next obtain, based on the occlusion metadata 305 and the location 317, a second renderer for a second group of audio objects 15F-15H, and apply the second renderer to the second group of objects to obtain second speaker feeds. The audio decoding device 24 may then obtain, based on the first speaker feeds and the second speaker feeds, the speaker feeds. More information regarding how physical occlusions, like a wall, may be defined via the occlusion metadata 305 is provided below with respect to the example of FIG. 4.
[0098] FIG. 4 is a block diagram illustrating an example occlusion and the accompanying occlusion metadata that may be provided in accordance with various aspects of the techniques described in this disclosure. As shown in the example of FIG. 4, an incident sound energy 470A (which may be denoted mathematically by the variable E.sub.i) represented by the audio data 15 may encounter an occlusion 472 (shown as a wall, which is one example of a physical occlusion).
