Apple Patent | Method and system for generating at least one image of a real environment

where P_tradeoff refers to the tradeoff parameters, P_vis refers to the ideal imaging parameters for visualization, P_pose refers to the ideal imaging parameters for camera pose determination, and alpha is a scalar weighting value in the range [0,1].

In an embodiment the tradeoff is computed using the maximal alpha which still enables pose determination. In an embodiment the method to determine the tradeoff parameter converges towards the desired parameter iteratively over a sequence of multiple images.

FIG. 7 illustrates another embodiment of the present invention in which a compromise (or tradeoff) between ideal imaging parameters for camera pose determination and ideal imaging parameters for visualization may be found and used for both camera pose determination and visualization. The augmented reality scene 700 in this case comprises a real marker 701 which is used for camera pose determination, a real cube 702 and a virtual sphere 703 which sits on the real cube 702 and should be visualized superimposed in an augmented image. The real environment further comprises of a real spotlight which illuminates the real marker 701. As a result, the marker is much brighter than the rest of the real environment including the real cube 702. This is clearly visible in 710, which illustrates a captured image of the real scene in which the imaging parameters are chosen such that the resulting image 710 is ideal for pose determination. In this case the exposure time is set such that the marker 711 is imaged contrasty to allow for proper recognition of the marker in the camera image for camera pose determination. The real cube 712, however, is imaged very dark and with very little contrast with this exposure time, which is not desirable for visualization.

Instead, an image of the scene captured with ideal imaging parameters for visualization is illustrated in 720 where the real cube 722 on which the virtual object should sit is well visible and contrasty. With these imaging parameters, however, the marker 721 is hardly visible and has a very low contrast because it is very bright and leads to oversaturation of the imaging sensor. As a result, image 720 is not well suited for camera pose determination. In one embodiment of the present invention, the image 710 is used for pose determination and image 720 is used for visualization. If the Real World Acquisition Apparatus however is not capable to provide images with different imaging parameters simultaneously, then another embodiment of the present invention can be employed. This embodiment finds a compromise between ideal imaging parameters for camera pose determination and ideal imaging parameters for visualization, for example based on the interpolation method mentioned above, and the resulting image 730 is used for both camera pose determination and visualization. In the present example, the marker 731 is not as contrasty as it would have been under ideal parameters for pose determination (711), but it has more contrast than 721 and therefore is suited for camera pose determination. The real cube 732 has less contrast than under ideal parameters for visualization (722), but it is much better visible than in image 710. Thereby, image 730 provides a tradeoff between images 710 and 720 which each are only well suited for either pose determination or visualization, while image 730 can be used for both.

In this example the value of the imaging parameters, e.g. the exposure time, can be determined based on the brightness of a part of the real environment in a camera image that was generated with known imaging parameters, particularly exposure time. As an example, if the part of the real object has an average intensity of 100 (in a range from 0 to 255) in an image generated with an exposure time of 7 milliseconds and the desired average intensity is 127, then the exposure time to be used to achieve the desired appearance could be computed as 7*127/100 milliseconds=8.89 milliseconds assuming a linear camera response. If the nonlinearities in the imaging system are known, e.g. gamma correction, then they can be considered accordingly by converting all intensity values to a linear space before applying the rule of proportion. The above process of determining a value of the imaging parameter exposure time can be performed repeatedly in an iterative fashion to converge to the desired appearance.

The brightness of a real environment in a (digital) image can be controlled by different means, which all may be used in embodiments of the present invention. Physically changing the aperture size or changing the shutter time or the exposure time affect the way light is measured in terms of brightness. After measurement, different means to process the measured data exist to change the brightness and contrast of images, e.g. adjusting levels, curves, brightness, contrast. A naïve implementation to increase the brightness in an image is to add a constant value to all intensities of the image or to multiply all intensities with a factor. Any combination of the above ways to change the brightness in an image of a real environment can be used by embodiments of the present invention to implement imaging parameters.

FIG. 8 illustrates another embodiment of the present invention, in which the brightness of a virtual object influences the exposure time of the Real World Acquisition Apparatus as an exemplary imaging parameter. Image 800 captures a real environment comprising a real cube 801. A standard approach for capturing such an image is the usage of the auto exposure feature of the camera. The camera meters how much light is coming from the environment, and sets an appropriate value for the exposure time and aperture. The camera in the example would detect that the room is “quite” dark, and therefore would set a longer exposure time in order to capture an image that has good contrast, and where details like the cube in the room are well visible. Image 810 shows the same image of the same environment comprising the real cube 811 superimposed with a virtual object 812, which is a virtual torch in this example. Image 810 does not look realistically because the virtual torch does not influence the visual appearance of the real environment comprising the real cube 811. If a real torch was added, it would not only illuminate the real scene, but the additional light from the bright torch towards the camera would also result in a decreased camera exposure time in case auto exposure is enabled, resulting in the real environment appearing darker. An embodiment of the present invention considers the virtual torch 822 by letting its brightness influence the exposure time of the real camera. Instead of determining the exposure time based only on the captured image as done in standard auto exposure approaches, an embodiment of this invention determines the exposure time, which is one imaging parameter, based on the captured image of the real environment and the virtual object, i.e. the virtual torch in this case. For example, an exposure meter (also called light meter) could be employed to detect the environment brightness or intensity of environment light. The data obtained from the exposure meter could be used together with the brightness of the virtual object (which could be provided as a property of the virtual object) in order to determine the exposure time for the camera capturing the real environment. In another example the brightness of the real environment is determined from a camera image of the real environment.

As can be seen in image 820 the exposure time of the real camera has been decreased to account for the virtual torch 822. This results in an increased realism of the augmented image. Note that in this example the virtual torch is not used to virtually illuminate the real scene. This could be done in an optional additional post processing step. The virtual object influences image parameters, such as the exposure time in this example.

FIG. 9 shows a further embodiment of the present invention. It comprises a Real World Acquisition Apparatus RWAA (901), a Pose Determination System (902), a Visualization System (903) and a Method IPDM V (904) determining imaging parameters for the image of the real environment used for the visualization. Additionally the system potentially comprises a Method IPDM PD (905) determining imaging parameters for the image of the real environment used for the pose determination.

The Pose Determination System receives (906) an image (907) of the real environment generated (924) by the RWAA (901) and determines (908) the pose (909) of the sensor apparatus based on this image (907). In this example, at least one of the pose (909) and the Virtual Object Description (911) is used (910) by the IPDM V (904) to determine (913) the Imaging Parameters (914), which are taken into account (917) by the Real World Acquisition Apparatus (901) for generating (918) an Image (919) of the Real Environment used for the Visualization. For determining (913) the Imaging Parameters (914) for the Image of the Real Environment used for the Visualization, the IPDM V (904) optionally considers (915) a Known Spatial Transformation (916) between the Pose (909) of the Sensor Apparatus where it captures the Image (907) of the Real Environment used for the Pose Determination and the Pose of the Sensor Apparatus where it captures the Image (919) of the Real Environment used for the Visualization. The IPDM V (904) optionally considers (931) the previously generated (918) Image (919) of the Real Environment used for the Visualization.

The generated Image (919) of the Real Environment then is used (920) by the Visualization System as the part of the augmented image showing the real environment which is overlaid by a rendering according (921) to the Pose (909) of the Sensor Apparatus of the Image (907) of the Real Environment used for the Pose Determination—potentially transformed (922) by the Known Spatial Transformation (916)—and according (923) to the Virtual Object Description (911).

The image (907) of the Real Environment used for the Pose Determination is generated (924) by the RWAA (901). Optionally there is a Method IPDM PD (905) which determines the Imaging Parameters for that image (932). In this case, the pose determination system (902) delivers (925) Pose Determination Characteristics (926), e.g. the current State of the Pose Determination, a history of the most current poses, and/or reconstructed 3D data of the scene. The Method IPDM PD (905) uses at least one of the characteristics (926), the pose (909), and the virtual object description (911) to determine (929) the Imaging Parameters (9032), which then are used (930) by the RWAA (901) to generate (924) the next image (907) of the Real Environment used for the Pose Determination.

FIG. 10 illustrates different setups (in the present example, three setups) related to embodiments of the present invention and describe how requested imaging parameters may be used for generating images. In the three setups Imaging Parameters are defined for an image for pose determination (1014, 1057, 1077) as well as Imaging Parameters are defined for an image for visualization (1008, 1058, 1078). The Imaging Parameters may influence the measurement step of the light leaving from the real environment, i.e. settings regarding the optics and sensors, as well as they may influence the further processing step of the measured data into the final images of the real environment, i.e. settings regarding the image processing.

The RWAA (1001) in FIG. 10a contains two separate Light Measure Apparatuses (1002, 1003). The Light Measure Apparatuses (1002, 1003) of RWAA (1001) may be implemented using two separate camera devices or as one camera device. The latter can be achieved by time scheduling (or time-division multiplexing) of light measurements for pose determination and light measurements for visualization. Each light measure apparatus (1002, 1003) may comprise at least one optics and sensor.

The RWAA (1001) in FIG. 10a contains two separate Process Apparatuses (1004, 1006). The Process Apparatuses (1004, 1006) may be implemented as separate apparatuses or as one apparatus. The latter can be achieved by for example time scheduling (or time-division multiplexing or interleaving) of processing of light measurements for pose determination and for visualization. The same is true for the Process Apparatuses (1053, 1055) in FIG. 10b.

The first light measure apparatus (1002) of the RWAA (1001) measures data about the light leaving from the real environment which then is processed by a process apparatus (1004) into an image (1005) used for pose determination. The second light measure apparatus (1003) of the RWAA (1001) separately measures data about the light leaving from the real environment which then is processed by a process apparatus (1006) into an image (1007) used for visualization. As two separate measurements about the light are taken, the measurements are independent of each other, and can be handled separately. Therefore the imaging parameters (1008) desired for the image for visualization are processed by a dedicated Imaging Parameters Manager IPMV (1009) as well as the imaging parameters (1014) desired for the image for pose determination are processed by a dedicated Imaging Parameters Manager IPMP (1015).

From the desired imaging parameters (1008) and according to the capabilities of the Light Measure Apparatus (1003) as well as Process Apparatus (1006) the IPMV (1009) determines which particular imaging parameters are used during measuring data about the light leaving from the real environment and which particular imaging parameters are used during the processing of the measured data into an image. The parameters determined to be used during measuring data are provided (1010) to and used by the Light Measure Apparatus (1003) for Visualization when measuring data about the light. These parameters could for example comprise a particular value for the focal length and exposure time. The data measured about the light leaving from the real environment according to the provided (1010) parameters then is transferred (1011) to the Process Apparatus (1006) responsible for the Image (1007) for Visualization.

The parameters determined by the IPMV (1009) to be used during processing the measured data into an image are provided (1012) to and used by the Process Apparatus (1006) responsible for the Image for Visualization when processing (1013) the measured data into a final image (1007) of the real environment used for visualization. These imaging parameters could for example comprise of particular values defining the amount of blurring of the image depending on depth and tone mapping for pixel intensities. If no imaging parameters (1008) are defined for a particular domain like for example focus or exposure, the light measure apparatus (1003) and process apparatus (1006) generating the image (1007) for visualization may exhibit default behavior, e.g. auto focus, auto exposure and auto white balancing.

Optionally also imaging parameters (1014) desired for the image (1005) used for pose determination are defined. Then from the desired imaging parameters (1014) and according to the capabilities of the Light Measure Apparatus (1002) as well as Process Apparatus (1004) the imaging parameters manager IPMP (1015) responsible for the image for pose determination determines which particular imaging parameters are used during measuring data about the light leaving from the real environment and which particular imaging parameters are used during the processing of the measured data into an image. The parameters determined to be used during measuring data are provided (1016) to and used by the Light Measure Apparatus (1002) for Pose Determination when measuring data about the light. These parameters could for example comprise of a particular value for the focal length and exposure time. The data measured about the light leaving from the real environment according to the provided (1016) parameters then is transferred (1017) to the Process Apparatus (1004) responsible for the Image (1005) for Pose Determination.

The imaging parameters determined by the IPMP (1015) to be used during processing the measured data into an image are provided (1018) to and used by the Process Apparatus (1004) responsible for the Image for Pose Determination when processing (1019) the measured data into a final image (1005) of the real environment used for pose determination. These imaging parameters could for example comprise of particular values defining the amount of blurring of the image depending on depth with a Gaussian kernel and tone mapping for pixel intensities.

If no imaging parameters (1014) are defined for a particular domain like for example focus or exposure, the light measure apparatus (1002) and process apparatus (1004) generating the image (1005) for the pose determination may exhibit default behavior, e.g. auto focus, auto exposure and auto white balancing.

The RWAA (1051) in FIG. 10b contains only a single Light Measure Apparatus (1052). It measures data about the light leaving from the real environment which then is processed (1064, 1066) separately by a process apparatus (1053) for pose determination into an image (1054) used for pose determination as well as by process apparatus (1055) for visualization into an image (1056) used for visualization. Thereby a measurement step of the light leaving from the real environment is shared between the image for Visualization and Pose Determination. The particular imaging parameters used (1060) for the measuring should be balanced between the imaging parameters (1057) requested for the image for pose determination and the imaging parameters (1058) requested for the image for visualization. Therefore both requested imaging parameters (1057, 1058) are processed by a combined Imaging Parameters Manager IPMC (1059).

From the desired imaging parameters (1057, 1058) and the capabilities of the Light Measure Apparatus (1053) as well as Process Apparatuses (1055, 1056) the IPMC (1059) determines which particular imaging parameters are used during measuring data about the light leaving from the real environment and which particular imaging parameters are used during the two separate processing steps of the measured data into the two images. The parameters determined to be used during measuring data should consider a good tradeoff between the two desired imaging parameters (1057, 1058) bearing in mind a separate processing step afterwards by 1053 and 1055. These imaging parameters could for example comprise of those particular values for the focal length and exposure time that result in measured data that allows to best fulfill the overall desired imaging parameters (1057, 1058) in the succeeding two separate processing steps. An example could be a first requested imaging parameter (1057) for overexposing sensor irradiance values above a threshold T1 and not overexposing sensor irradiance values below T1 as well as a second requested imaging parameter (1058) for overexposing sensor irradiance values above a threshold T2 and not overexposing sensor irradiance values below T2. The imaging parameters for the light measurement then are determined to achieve a measurement where at least sensor irradiance values up to the value of TMAX, where TMAX is the maximum of T1 and T2, are not overexposed. The imaging parameters for the light measurement are chosen in that way so that imaging parameters for the tone mapping in the particular processing steps of the measured data into the particular images then can be set accordingly to achieve an overexposure for sensor irradiance values above T1 in the first image (1054) as well as above T2 in the second image (1056) without overexposing sensor irradiance values below T1 and T2 in the images respectively.

The parameters determined by the IPMC (1059) to be used during measuring data about the light are provided (1060) to and used by the Light Measure Apparatus (1052) when measuring data about the light. The data measured about the light leaving from the real environment according to the provided (1060) imaging parameters then is transferred (1061, 1062) both to the Process Apparatus (1053) responsible for the Image for Pose Determination and to the Process Apparatus (1055) responsible for the Image for Visualization.

The imaging parameters determined by the IPMC (1059) to be used during further processing the measured data into the final image (1054) of the real environment for pose determination depend on the original desired imaging parameters (1057) for the image, the imaging parameters (1060) used for measuring the light as well as the capabilities of the process apparatus. These imaging parameters could for example comprise of particular values defining the amount of blurring of the image depending on depth with a Gaussian kernel and tone mapping for pixel intensities. In the above example, the imaging parameters for the tone mapping when processing the measured data into the image (1054) for pose determination would be set so that sensor irradiance values above T1 would be overexposed in the image (1054) while sensor irradiance values below T1 would not be overexposed. These imaging parameters are provided (1063) to and used by the Process Apparatus (1053) responsible for the Image for Pose Determination when processing (1064) the measured data into a final image (1054) of the real environment used for pose determination.

The imaging parameters determined by the IPMC (1059) to be used during further processing the measured data into the final image (1056) of the real environment for visualization depend on the original desired imaging parameters (1058) for the image, the imaging parameters (1060) used for measuring the light as well as the capabilities of the process apparatus. These imaging parameters could for example comprise particular values defining the amount of blurring of the image depending on depth with a Gaussian kernel and tone mapping for pixel intensities. In the above example, the imaging parameters for the tone mapping when processing the measured data into the image (1056) for visualization would be set so that sensor irradiance values above T2 would be overexposed in the image (1056) while sensor irradiance values below T2 would not be overexposed. These imaging parameters are provided (1065) to and used by the Process Apparatus (1055) responsible for the Image for Visualization when processing (1066) the measured data into a final image (1056) of the real environment used for visualization.

The RWAA (1071) in FIG. 10c contains only one Light Measure Apparatus (1072). It measures data about the light leaving from the real environment which then is processed by a single process apparatus (1075) for pose determination and visualization into an image (1076) used both for pose determination as well as for visualization. The image (1076) thus is shared for Pose Determination and Visualization. The particular imaging parameters used for the measuring and processing should be balanced between the two sets of imaging parameters (1077, 1078) separately requested—the imaging parameters (1077) requested for the image for pose determination and the imaging parameters (1078) requested for the image for visualization. Therefore both requested imaging parameters (1077, 1078) are processed by a combined Imaging Parameters Manager IPMC (1079).

From the desired imaging parameters (1077, 1078) and according to the capabilities of the Light Measure Apparatus (1072) as well as of the Process Apparatus (1075) the IPMC (1079) determines which particular imaging parameters are used during measuring data about the light leaving from the real environment and which particular imaging parameters are used during the processing of the measured data into the image. The parameters determined to be used during measuring data as well as the parameters determined to be used during the processing step should consider a good tradeoff of their composite effect for the final image (1076) when compared with the two desired imaging parameters (1077, 1078). These imaging parameters could for example comprise of those particular values for the focal length and exposure time that result in measured data that allows to best fulfill the overall desired imaging parameters (1077, 1078) considering potential adjustments in the succeeding processing step. An example could be a first requested imaging parameter (1077) that specifies that a particular distance D1 should be in focus as well as a second requested imaging parameter (1078) that specifies that a particular distance D2 should be in focus. The imaging parameters for the light measurement then are determined to achieve a measurement where both points in the real environment in a distance of D1 from the camera are mapped sufficiently sharp as well as points in the real environment in a distance of D1 from the camera are mapped sufficiently sharp. This can for example be achieved by influencing the depth of field to be sufficiently large to cover both distances by setting a small enough aperture value.

The parameters determined by the IPMC (1079) to be used during measuring data about the light are provided (1080) to and used by the Light Measure Apparatus (1072) when measuring data about the light. The data measured about the light leaving from the real environment according to the provided (1080) imaging parameters then is transferred (1082) to the Process Apparatus (1075) responsible for the Image (1076) used for Pose Determination and Visualization.

The imaging parameters determined by the IPMC (1079) to be used during further processing the measured data into the final image (1076) of the real environment depend on the two original desired imaging parameters (1077, 1078) for the image, the imaging parameters (1080) used for measuring the light as well as the capabilities of the process apparatus (1075). These imaging parameters could for example comprise of particular values defining the amount of blurring of the image depending on depth with a Gaussian kernel and tone mapping for pixel intensities. In the above example, there could for example be another request in one of the requested imaging parameters (1077, 1078) that points closer than a distance D3 should be out-of-focus. This out-of-focus effect could maybe not be achieved in the light measurement while at the same time keeping distances D1 and D2 in focus. The imaging parameters for the blurring of the image then could be set in such a way, that parts of the image closer than D3 are blurred with a Gaussian kernel with the amount of blurring depending on the depth a.k.a. distance of that part from the camera. Other examples could be tone mapping with a mapping function that is a tradeoff between the requested imaging parameters (1077, 1078). These imaging parameters are provided (1085) to and used by the Process Apparatus (1075) responsible for the Image for both Pose Determination and Visualization when processing (1082) the measured data into the final image (1076) of the real environment.

Note that often not all requested imaging parameters can be fulfilled because of limitations of capabilities of the light measure and process apparatuses. Also requests regarding imaging parameters may conflict. Thereby either a single set of requested imaging parameters can be inconsistent within itself or multiple requested imaging parameters may conflict. In these cases, those imaging parameters should be chosen that minimize the introduced error in the effect visible in the images. The error can be for example measured by giving different imaging parameters weights and by defining a metric on the parameter space, that allows to measure the deviation between the required imaging parameters and the used once by a scalar.

In another embodiment the metric is defined over the space of image properties, like for example image brightness or radius of the circle of confusion, which result from at least one of imaging parameters like exposure time, aperture, tone mapping, focal length and depth of field. The error regarding one image property then can be for example measured by the difference between the image property which would result from the requested imaging parameters and the image property resulting from the chosen imaging parameters. The overall error can be defined for example as sum over the absolute errors or the square root of the sum over the squared errors of the particular image properties. Giving different image properties weights allows tuning the influence and importance of a particular image property.

The method described herein deals with different types of imaging parameters. On the one hand the imaging parameters manager of the pose determination system and the imaging parameters manager of the visualization system specify particular imaging parameters for an intended image to be generated, like a focus distance at 2 meter and a depth of field of approximately 3 m.

As an example, the imaging parameters manager of the visualization system could for example first determine whether at the center part of the planned augmented image a part of the real world will be visible or if this part of the real world will be overlaid by a rendering of a virtual object. This could be checked based on the geometry (also known as virtual object property) of the virtual object and the camera pose (either provided or determined from a provided image of the environment) by sampling the center pixel by terms of, for example, ray tracing or by sampling from a rasterized image of the virtual object or of its bounding volume. If a virtual object is found for that pixel, the sampled distance of the virtual object potentially can be compared against the real world depth for that pixel, for example based on a depth camera image, a depths reconstruction or a provided model of the real world.

If the visualization system determines that the real world will be visible in the center of the planned augmented image, the visualization system could for example request a focus distance corresponding to the distance between the camera and the part of real world visible at the center of the image. The information about the distance could be determined from a (partial) 3D model of the scene provided by for example a depth camera or a reconstructed or provided 3D model of the scene together with the current camera pose.

If the visualization system determines that a virtual object will be visible in the center of the planned augmented image, the visualization system could for example request a focus distance corresponding to the distance between the virtual camera and the part of the virtual object visible at the center of the image. The information about the distance could be determined from the 3D model (also known as virtual object property) of the virtual object and the current camera pose (either provided or determined from a provided image of the environment).

It is of course also possible to not only examine one but multiple locations of the planned augmented image, and then further process the different distances for example by calculating a weighted average over the different distances. Also it would be possible to not examine a particular location by means of raytracing or rasterization, but to simply calculate the distance of a virtual object or some point of the virtual object to the camera, and use this information to set the imaging parameters for the focus distance.

The imaging parameters manager of the pose determination system could for example determine at what distance to the camera there are those features in the real world that will lead to feature points in the camera image that are well suited for pose determination. This could for example be identified based on the actual camera pose and or the reconstructed point cloud or CAD model of the real world object which is used for the pose determination. The pose determination system then could for example request a focus distance and depths of field corresponding to those distances between the camera and the part of real world, so that the wanted features are mapped in focus and sharply.

Beside focus, there are also other imaging parameters, which are beneficial to influence. Talking about exposure, the visualization system could for example request that the luminance values of a particular part of the virtual object which is visible in the planned augmented image should be well exposed (not over- or underexposed). The particular part of the virtual object visible in the planned augmented image would be determined by the current camera pose. For this particular part (or the whole object) the luminance values could be determined, either by direct specifications of luminance in the virtual object properties or by shading calculations, that is a combination of the virtual object surface properties and the virtual object illumination property. The pose determination system could for example request that the luminance value of a particular part of the real world used for pose determination should be well exposed (not over- or underexposed). The particular part of the real world and the corresponding luminance of that part could be determined based on a CAD model of the real world object used for the pose determination and the current camera pose. The corresponding luminance information could also stem from a previous reconstruction of the real world object, which includes reconstruction of luminance.

In the same way, it could also be beneficial to determine imaging parameters, so that the full (or another specified part of the) dynamic range of the planned image is used by a certain range of luminance values specified by a virtual object. This then can be either achieved by adequate imaging parameters during the light measuring step or during the succeeding processing.

In the same way, it could also be beneficial to determine imaging parameters, so that the resulting image of the real world used for visualization contains a certain amount of motion blur. This could for example be specified by a virtual object depending on the speed of the virtual object and the expected speed of camera motion in order to keep the virtual object in target. A desired amount of motion blur (for example 15 pixels) then could be achieved by appropriately adapting the exposure time.

On the other hand there may be a particular implementation of a real world acquisition apparatus, which will be used to generate the intended image. Each particular implementation of such an apparatus has its own physical limitations and technical capabilities in terms of for example the used system of lenses, lens distortions, mirrors, field of view, optical zoom range, focus distances, depth of field, aperture stops and shape, focal length, focal planes, vignetting, sensor type, sensor size, bit-depth, dynamic range, resolution, sensor gain, color channels, color filters, shutter speed and lag, read out times, processing power, sensor bias and its correction, defect pixels and its correction, dark frame substraction, white balance, demosaicing, denoising, sharpening, color space conversion, linearity, gamma correction, image filters, noise characteristics and reductions, or employed algorithms and compression. Each particular implementation also may vary in the degree of configurability of the involved pipeline as well as the possibility to query particular settings and calibrate its behaviour. Additionally also the possibility of an apparatus to determine for example the optimal exposure, white balance or focus setting may differ depending on for example the availability of an additional auto-focus sensor or a metering sensor for auto-exposure, auto-white-balance or motion prediction.

According to an embodiment, the generation of an image may comprise at least two steps. A first step is the measurement of light intensities by a sensor, which also includes optical components that influence what particular part of the light field is guided to a particular sensor position. A second step is the succeeding processing of the measured data into the final image of the real world.

There is not always a one to one mapping between a requested imaging parameter for an intended image and the imaging parameter to be finally set at the real world acquisition apparatus.

Exposure for example is the amount of light per unit area reaching the image sensor and depends on shutter speed, exposure time, lens aperture, as well as scene luminance. To achieve a particular exposure value, different parts of the light measurement step (shutter speed or lens aperture) could be adjusted. For a static real world scene and camera however the exposure corresponding to a twice as long exposure time could also be simulated by just doubling the exposure values.

A focus distance can be achieved by either changing the distance between sensor and lens, or by changing the behavior of the lens (respectively system of lenses) itself. A depth of field can be achieved by a particular focal length and aperture. The aperture however again also influences the exposure.

By this it should be clear that parts of the desired effects of the requested imaging parameters can be achieved by taking influence on one or multiple imaging parameters in a step one, the measurement of light, others by taking influence on or multiple imaging parameters in a step two, the processing of the data, and some effects can even be achieved by taken influence in either one of the step one or two or in both simultaneously. Information which is not measured in the first step, can however not be produced by processing the measurements in step two.

A Light field camera like the so-called Lytro for example has a particular so-called “refocus range” which is the distance range between objects in the foreground and background that can be later on re-focused by processing the measured data into images. This refocus range is determined by the settings used during the measurement of light. Although the particular focus distance—for example F1=2 meter—then is selected during processing, the settings used for the measurement should be set accordingly to allow the particular focus distance—in this example F1.

A similar thing would be a particular tone mapping of an intended image, which should for example overexpose sensor irradiance values above a threshold T1, and not overexpose sensor irradiance values below T1. This could be achieved by only setting particular imaging parameters for the light measurement. It can however also be achieved by measuring a higher dynamic range well exposed (i.e. without any overexposure, saturation, or intensity clipping), and afterwards apply tone mapping to synthetically overexpose values above the threshold T1. If the range below T1 however is already overexposed during measuring, tone mapping during processing cannot undo the overexposure, i.e. intensity clipping.

If a particular real world acquisition apparatus cannot achieve a particular imaging parameter, or a particular combination of imaging parameters (for example combination of shutter speed and exposure, or the combination of focus distance and depth of field), the imaging parameters best meeting the intended effect should be found. As an example, if the closest focus distance for a real world acquisition apparatus is F2=0.3 meter and the intended imaging parameters would request a focus distance of F3=0.2 meter, the focus distance of the real world acquisition apparatus should be set to F2, the closest approximation or nearest neighbor of the intended parameter.

If not only one, but at least two images shall be generated based on a single light measurement step, the finally used imaging parameters for the light measurement step should be set to best satisfy the needs for all the images. Thereby it could also be beneficial to weight the importance of the different imaging parameters, to for example ensure that at least the pose determination on the one image works out. Similarly one could give a weight to a particular imaging parameter, for example specifying that focus distance is more important than the depth of field.

One example for a shared light measurement step could be a real world acquisition apparatus comprising of a light field camera and a first requested imaging parameter for the focus distance of F1=2 meter as well as a second requested imaging parameter for the focus distance of F2=5 meter. The imaging parameters for the light measurement then are determined to best allow a refocus range including at least F1 as well as F2. The imaging parameters for the particular processing of the measured data into the particular images then are set accordingly to achieve a final focus distance of F1 and F2 respectively.

This is for example shown in FIG. 10b, where the imaging parameters 1057 comprise a focus distance of F1 and the imaging parameters 1058 comprise a focus distance of F2. The imaging parameters provided (1060) to the light measure apparatus (1052) would in this case comprise the specification that the range of focus distances for which images can be processed from the acquired data includes F1 and F2. After acquisition, process apparatus 1053 would process the data into an image with a focus distance of F1 and process apparatus 1055 would process the data into an image with a focus distance of F2. This can be achieved with so-called “post focusing”, which assembles images with a particular focus from light rays stored in a light field dataset.

In another embodiment according to FIG. 10b, the Real World Acquisition Apparatus is a monocular camera allowing for control over the focus. Provided with first imaging parameters 1057 comprising a focus distance of F1 and second imaging parameters 1058 comprising a focus distance of F2 the Imaging Parameters Manager may determine a compromise focus distance F3 e.g. as the average of F1 and F2, i.e. F3=0.5*F1+0.5*F2 or any other weighted combination of the two distances, i.e. F3=(1−a)*F1+a*F2 with a in the range [0,1]. The imaging parameters provided (1060) to the light measure apparatus (1052) would in this case comprise a focus distance of F3. The imaging parameter is then considered by the light measure apparatus e.g. by mechanically changing the position of lenses or other optical elements or by mechanically changing the shape of optical elements. After acquisition, process apparatus 1053 would process the data (with a focus distance of F3) into an image with a focus distance of F1 and process apparatus 1055 would process the data (with a focus distance of F3) into an image with a focus distance of F2 by means of convolution and deconvolution.

Since the amount of defocus of a part of the real environment present in an image depends on focus distance and the distance of the imaged part of the real environment, changing focus-related imaging parameters with the process apparatus may require knowledge on the distance of parts of the real environment to the camera to treat them according to their distance and to change their amount of defocus (or blurriness) accordingly.

To increase the amount of defocus in a particular part of the image using a process apparatus, one approach is to convolve the part of the image with a Gaussian kernel, which is a two-dimensional normal distribution and mimics the point spread function of an imaged point in the real environment on the image. Analogously to decrease the amount of defocus in a particular part of the image using a process apparatus, one approach is to deconvolve the part of the image with a Gaussian kernel.

In this embodiment, the imaging parameters referring to focus distance were partially implemented by imaging parameters affecting the measurement of light (e.g. changing the lens position), and partially implemented by imaging parameters affecting the processing of the measurements into final images (e.g. blurring with a Gaussian kernel). In the processing of the measurements into images, one or more intermediate images may be generated and the one or more intermediate images may be convolved or deconvolved with a Gaussian kernel in order to produce the final images.

Many other imaging parameters besides the focus distance can also be implemented in the measuring of light, or in the processing of measurements into images, or as a combination of both. The processing of the measurements into images may include generating one or more intermediate images, and the generating one or more intermediate images may be further processed to become final images according to at least one of the imaging parameters.

Another example could be a real world acquisition apparatus with a first requested imaging parameter for overexposing sensor irradiance values above a threshold T1 and not overexposing sensor irradiance values below T1 as well as a second requested imaging parameter for overexposing sensor irradiance values above a threshold T2 and not overexposing sensor irradiance values below T2. The imaging parameters for the light measurement then are determined to achieve a measurement where at least sensor irradiance values up to the value of TMAX, where TMAX is the maximum of T1 and T2, are not overexposed. The imaging parameters for the particular processing of the measured data into the particular images then are set accordingly to achieve an overexposure for sensor irradiance values above T1 and T2 respectively without overexposing sensor irradiance values below T1 and T2 respectively.

Requested imaging parameters for a desired image may not only comprise a single value, but also a range of possible values, potentially equipped with a rating of suitability for the values, which then can be used within the process of finding the optimal compromise imaging parameters. Additionally it is also possible to have multiple requested imaging parameters by different systems but only generate one image which then is used in multiple ways, for example for pose determination as well as visualization based on the multiple requested imaging parameters from different imaging parameter managers. Here again optimal compromise imaging parameters should be determined for the light measurement step as well as the processing step, that best satisfy the needs from the multiple requested imaging parameters.

FIG. 11 shows a data flow for a system comprising two separate Light Measure Apparatuses. The left half of the Figure relates to generating images of the real environment and determining the pose of the camera according to these images, the right half of the Figure relates to generating images of the real environment used for visualization of the real environment in an augmented image. In both parts imaging parameters for the particular images are determined.

At the beginning (1101) there is not yet a determined previous pose (1102), so the imaging parameters (1105) for the image used for determining the pose are set (1104) to initial image parameters based on characteristics of the pose determination method or manually provided. If later on, when there (1102) is already a previous pose, the imaging parameters (1105) for the image used for determining the pose are determined (1116) based on previous poses and optionally based on characteristics of the pose determination method. Characteristics of the pose determination method could for example include a preferred focal distance e.g. that only parts of the real environment should be imaged sharply that are close to the camera or that only parts of the real environment should be imaged sharply that are far away from the camera or that all parts of the environment should be imaged sharply. Other characteristics of the pose determination method could include requirements on the sharpness of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the respective part should be imaged sharply with a point spread function smaller than a pixel. Other characteristics of the pose determination method include requirements on the contrast of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the difference between the intensity of the brightest pixel and the intensity of the darkest pixel is greater than a given threshold.

Based on the imaging parameters (1105) an image (1107) is generated (1106) of the real world and a pose (1109) is determined (1108) from that image (1107). This pose (1109) then is used again for determining (1116) imaging parameters (1105) for the next image used for determining the pose.

At least one of the pose (1109) (and potentially additional previous poses) and virtual object properties (1111) are used to determine (1110) imaging parameters (1112) for the image used for visualization. Based on the imaging parameters (1112) an image (1114) is generated (1113) of the real world and the image (1114) then is used (1115) for visualization in an augmented image.

Different implementations of the pose determination method or even a single implementation in different states may have particular preferences about the properties of the images used for pose determination, referred to as pose determination method characteristics.

In an embodiment pose determination method characteristics include a preferred focal distance e.g. that only parts of the real environment should be imaged sharply that are close to the camera or that only parts of the real environment should be imaged sharply that are far away from the camera or that all parts of the environment should be imaged sharply.

In another embodiment pose determination method characteristics include requirements on the sharpness of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the respective part should be imaged sharply with a point spread function smaller than a pixel.

In another embodiment pose determination method characteristics include requirements on the contrast of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the difference between the intensity of the brightest pixel and the intensity of the darkest pixel is greater than a given threshold.

FIG. 12 shows a similar data flow as FIG. 11, however, for an embodiment that only comprises one Light Measure Apparatus. At the beginning (1201) there is not yet a determined previous pose (1202), so the desired imaging parameters (1205) for the image used for determining the pose are set (1204) to initial image parameters based on characteristics of the pose determination method or manually provided. If later on, when there (1202) is already a previous pose, the desired imaging parameters (1205) for the image used for determining the pose are determined (1216) based on previous poses and optionally based on characteristics of the pose determination method.

Characteristics of the pose determination method could for example include a preferred focal distance e.g. that only parts of the real environment should be imaged sharply that are close to the camera or that only parts of the real environment should be imaged sharply that are far away from the camera or that all parts of the environment should be imaged sharply. Other characteristics of the pose determination method could include requirements on the sharpness of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the respective part should be imaged sharply with a point spread function smaller than a pixel. Other characteristics of the pose determination method include requirements on the contrast of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the difference between the intensity of the brightest pixel and the intensity of the darkest pixel is greater than a given threshold.

If there is once at least one pose (1209) available, at least one of the pose (1209) (and potentially additional previous poses) and virtual object properties (1211) are used to determine (1210) desired imaging parameters (1212) for the image used for visualization. Now as there is only one Light Measure Apparatus in this embodiment, the two desired imaging parameters (1205, 1212) are combined (1226) into compromise imaging parameters (1227), bearing in mind a shared light measuring and a separate processing afterwards. Based on the imaging parameters (1227) two images are generated (1228) of the real world, one image (1207) for pose determination, which then successively is used to determine (1208) the pose (1209), another image (1214) for visualization, which then is used (1215) as real world part for a later on augmented image. Thereby images 1207 and 1214 may be the same image or they may be based on the same light measurement but differ in terms of processing applied to the measured light to generate the respective images 1207 and 1214.

Initially, when there is not yet a pose available, the compromise imaging parameters (1227) are either determined (1226) based on imaging parameters (1205) for the image for pose determination only, or some default for the imaging parameters (1212) for the image used in visualization are set.

FIG. 13 shows a similar data flow as FIG. 12, with an embodiment in which the system comprises only one Light Measure Apparatus. In this case however, the parameters for image processing for the image for visualization are updated on the current pose determined by the measured light data, which pose and light data then is also used for that particular image for visualization.

At the beginning (1301) there is not yet a determined previous pose (1302), so the desired imaging parameters (1305) for the image used for determining the pose are set (1304) to initial image parameters based on characteristics of the pose determination method or manually provided. If later on, when there (1302) is already a previous pose, the desired imaging parameters (1305) for the image used for determining the pose are determined (1316) based on previous poses and optionally based on characteristics of the pose determination method.

Characteristics of the pose determination method could for example include a preferred focal distance e.g. that only parts of the real environment should be imaged sharply that are close to the camera or that only parts of the real environment should be imaged sharply that are far away from the camera or that all parts of the environment should be imaged sharply. Other characteristics of the pose determination method could include requirements on the sharpness of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the respective part should be imaged sharply with a point spread function smaller than a pixel. Other characteristics of the pose determination method include requirements on the contrast of the image of the part of the real environment relative to which the camera pose shall be determined, e.g. that the difference between the intensity of the brightest pixel and the intensity of the darkest pixel is greater than a given threshold.

If there is once at least one pose (1309) available, at least one of the pose (1309) (and potentially additional previous poses) and virtual object properties (1311) is used to determine (1310) desired imaging parameters (1312) for the image used for visualization of the real environment based on the upcoming light measurement. Now as there is only one Light Measure Apparatus in this embodiment, the two desired imaging parameters (1305, 1312) are combined (1330) into compromise imaging parameters (1331) for the light measurement, bearing in mind a shared light measuring and a separate processing afterwards.

According to the compromise imaging parameters (1331) light of the real environment is measured (1332). This light measurement (1333) data of the real world then is processed (1334) into an image (1307) for pose determination according to the desired imaging parameters (1305) for the image for pose determination, considering the applied compromise imaging parameters (1331) for measuring (1332) the light.

The image (1307) for pose determination then is used to determine (1308) the latest pose (1309). This latest pose (1309) (and potentially additional previous poses) is used together with virtual object properties (1311) to determine (1310) desired imaging parameters (1312) for the image used for visualization of the real environment based on the pose determined from the newest light measurement. The newest light measurement however is already made, so only the processing part can still be influenced. Therefore the newest desired imaging parameters (1312) for the image used for visualization of the real environment based on the newest light measurement are compared to the imaging parameters (1331) used for the measurement of the newest light measurement. This is done by an imaging parameters updater (1339) which outputs updated parameters for the processing of the current light measurement into the image visualization considering both the imaging parameters (1331) used for the newest light measurement and the imaging parameters (1312) based on this newest light measurement as well as the virtual object properties (1311). These updated imaging parameters (1340) determine how the already measured light data (1333) shall be processed into an image for visualization and consider the used imaging parameters for the light measurement (1331). The light measurement (1333) data of the real world then is processed (1341) into an image (1314) for visualization according to the updated imaging parameters (1340) for the image for visualization, considering the applied compromise imaging parameters (1331) for measuring (1332) the light. The resulting image (1314) then is used (1315) for visualization in an augmented image.

Initially, when there is not yet a pose available, the compromise imaging parameters (1331) are either determined (1330) based on imaging parameters (1305) for the image for pose determination only, or some default for the imaging parameters (1312) for the image used in visualization are set.

FIG. 14 explains a terminology of generating images of the real environment using specified imaging parameters according to embodiments of the invention.

Particularly, FIG. 14a shows a simplified case, according to specified imaging parameters (1401) for an image, an image (1403) is generated (1402) of the real world or environment.

FIG. 14b illustrates generating an image of the real world comprising at least two steps: a light measuring step and a step of processing the measured data into an image. Therefore the specified imaging parameters (1404) for the image are split (1405) into separate imaging parameters (1406) for the light measuring step and imaging parameters (1407) for the processing step of the measured data into the final image. Based on the imaging parameters (1406) for the light measuring step data (1409) of the light of the real world is measured (1408). This measured data of the real world (1409) then is processed (1410) into an image (1411) of the real world according to the imaging parameters (1407) for the data to image processing.

FIG. 14c illustrates how the generation of multiple images may share a light measurement step. Two different imaging parameters may be specified. Firstly the imaging parameters (1412) for an image 1, and secondly the imaging parameters (1413) for an image 2 are provided. The generation of the two images will share a light measurement step, but will potentially have separate processing steps of the measured data into the final images. Therefore the specified imaging parameters (1412, 1413) for the images are transformed (1414) into combined imaging parameters (1415) for the light measuring and two separate imaging parameters (1416, 1417) for the data to image processing for image 1 and the data to image processing for image 2. Based on combined imaging parameters (1415) for the light measuring step, data (1419) of the light of the real world is measured (1418). This measured data of the real world (1419) then is processed twice. Firstly it is processed (1420) into an image 1 (1421) according to the imaging parameters (1416) for the data to image processing for image 1. Secondly it is processed (1422) into an image 2 (1423) according to the imaging parameters (1417) for the data to image processing for image 2.

According to this embodiment, the resulting image 1 (1421) should ideally be close to fulfilling the requirements specified with the imaging parameters 1412 and the resulting image 2 (1423) should ideally be close to fulfilling the requirements specified with the imaging parameters 1413. This ultimately depends on how well 1412 and 1413 are compatible.

Note that there are even more ways to combine and split the parameters not illustrated in this Figure, as for example visualized in FIG. 13, where the parameters for image processing for the image for visualization are updated on the current pose determined by the measured light data, which then is also used for the image processing for the image for visualization.

If a compromise for the imaging parameters for the image used for pose determination and the imaging parameters for the image used for visualization is used, it is also possible to use the same image for pose determination and visualization.

FIG. 15 illustrates different embodiments of the present invention related to a way how light may be measured by the real world acquisition apparatus (or part of it) to determine imaging parameters based on the appearance of the real environment. The imaging parameters that can be determined this way for example include shutter speed or aperture size. The figure explains how this metering can be combined with measurements for the lighting conditions introduced by the virtual object. The metering thereby is not limited to light intensities, but can also be used when measuring depth for determining imaging parameters like focus. For depths, the intensities in the depicted images (1501, 1502, 1503, 1504) can be understood as depths values, with white being far away from the camera and the darker the closer a particular region in the image is to the camera.

Image 1501 shows a view of the real environment comprising a mountain panorama view with a bright very distant sky (1507), a medium bright skyline (1505) of distant mountains as well as a dimmer stony ground (1506) quite close to the camera. Different options exist for measuring the light, for example by assigning a relative weighting to different light regions. A point in the image near the center of the image or near a particular point in the image defined as measurement center could have a higher weighting then a point further away from the measurement center, and thus could contribute more to the final exposure calculation. Different modes like center-weighted, partial metering or spot metering exist in nowadays cameras which differ in the degree of decline of weighting with respect to distance and the corresponding percentage of covered image region. Also the location of autofocus and different properties of particular regions in the image like color or contrast can contribute to the weighting.

Cameras often have particular measurement zones defined in the image, where they measure values like intensity, contrast, sharpness or depth for determining settings for auto focus or auto exposure. In the example shown in Image 1501, the measurement zones (1508) comprise a 3×3 grid of quads in the center of the image with two additional rectangles to the left and right. In these measurement zones, for example the light intensity is metered and the measurements are combined to determine imaging parameters like the shutter speed and aperture. In Image 1501 the top row would meter the bright intensity from the sky (1507), the medium row would (mainly) meter the medium bright light coming from the medium bright skyline (1505) and the bottom row would mainly meter the dim close by ground (1506), with the bottom left zone being little bit brighter, as also partially being covering the medium bright skyline (1505). The camera could then for example determine the real environment to be medium bright in average, and accordingly set imaging parameters like the shutter speed and aperture so that the resulting image is neither underexposed nor overexposed.

For augmented reality however, the imaged scene in the augmented image comprise not only the real environment but also virtual objects. The calculation of imaging parameters like exposure or focus should therefore also incorporate the appearance of the later on augmented virtual objects.

Based on virtual object properties, like the geometry of a virtual object, and the current camera pose, those measurement zones can be determined, which will be affected by the rendering of a virtual object. Image 1502 illustrates those zones (1521) depicted in bold, which cover a virtual object (1520), i.e. a dark sphere close to the camera. Zones (1522) which do not cover the virtual object (1520) are depicted with light lines.

In an embodiment illustrated in image 1503, only those measurement zones (1532) are used to find the best focus or exposure proposal for the real environment image which will not be overlaid by a rendering of a virtual object in the final augmented image, depicted in bold in image 1503.

In an embodiment illustrated in image 1502, measurement zones (1520) which cover a rendering of a virtual object in the final augmented image, depicted in bold in image 1502, are used to find the best focus or exposure proposal for the virtual object. The particular influence of a virtual object on the measurements and the determined proposal values for imaging parameters may depend on the provided virtual object properties.

These proposal values from the zones corresponding to the real environment part then can be combined with the proposal values from the zones corresponding to the virtual objects according to the influence of the particular measurement zones.

In another embodiment the proposal values corresponding to the real environment part use all measurement zones, independent from the virtual object, as depicted in image 1501.

In another embodiment, the proposal values corresponding to the virtual object are determined by other means based on for example the distance of the virtual object to the camera, the viewing direction between the camera and the virtual object or based on the image area covered by the virtual object.

In an embodiment the final imaging parameters may either be determined by combining determined imaging parameters from real and virtual measurements like described above, potentially using real world measurement zones according to occlusion of the real environment by virtual object.

In an embodiment the final imaging parameters for exposure may also be determined by making a combined measurement on an augmented image like shown in image 1504, where the augmented image potentially comprises a combined intensity image to determine exposure parameters. The virtual object is potentially rendered with matched imaging parameters for exposure with regard to the image of the real environment.

In an embodiment the image used to determine expose parameters has a High Dynamic Range.

In an embodiment the final imaging parameters for focus may also be determined by making a combined measurement on an augmented image like shown in image 1504, where the augmented image potentially comprises a combined depth map of the real environment and the virtual objects.

In an embodiment the imaging parameters for exposure are adjusted according to a brightness histogram of at least part of at least one image.

In an embodiment the exposure is adjusted in a way to try to create an augmented image with a histogram whose primary peak is in the midtones. In an embodiment the exposure is adjusted in a way to try to create an augmented image with a histogram whose primary peak is in the highlights. In an embodiment the exposure is adjusted in a way to try to create an augmented image with a histogram whose primary peak is in the shadows. In an embodiment the exposure is adjusted in a way to try to create an augmented image with a histogram whose primary peak is broad.

Further embodiments are disclosed in the following without explicit reference to the drawings.

Imaging parameters and determination thereof:

In an embodiment of the present invention, imaging parameters may include, but are not limited to, at least one or more of: focus, post focus, focus settings, lens position, focal plane, depth of field, focus distance, depth of focus, focus spread, sharpening, blurring, brightness, contrast, exposure time, shutter time, shutter mode, shutter settings, exposure settings, aperture, film speed, sensitivity, gain, ISO, low light settings, point of view, viewing direction, camera projection properties, focal length, field of view, zoom settings, region of interest, image cropping, extraction of picture elements, subject area, tone mapping, white balance, saturation, gamma correction, black level adjustment, quantization, filtering, noise reduction, edge enhancement, contrast enhancement, false chroma suppression, chroma subsampling, chroma resampling, color transformation, image compression, debayering, color space conversion, demosiac, blooming, convolution, deconvolution, filtering, NPR, toon shading, half toning, silhuette rendering, compensation, filter pattern compensation, rolling shutter compensation, sensor offset bias correction, pixel defects correction, lens shading (vignetting) compensation, dark frame subtraction, dynamic range, color format, bit depth, frame rate, resolution, flash settings, torch settings.

In a further embodiment imaging parameters are provided as pairs of parameter types and respective values, wherein the values can be indicative of a scalar value, a real-valued vector, a real-valued matrix, a range, a distribution, or a function.

In a further embodiment, imaging parameters are focus settings, such as lens position, focal plane, depth of field, focus distance, depth of focus, and focus spread.

In a further embodiment, imaging parameters are related to brightness, such as contrast, exposure time, shutter time, shutter mode, shutter settings, exposure settings, aperture, film speed, sensitivity, gain, ISO, low light settings.

In a further embodiment, imaging parameters are focus settings or parameters related to brightness.

In an embodiment a method for determining imaging parameters for visualization considers the distance of a virtual object to a camera and determines an imaging parameter which sets the focus distance to the distance of the virtual object.

In a further embodiment a method for determining imaging parameters for visualization considers the minimal and maximal distance of a virtual object to a camera and determines an imaging parameter which sets the depth of field to the difference between the minimal distance and the maximal distance of the virtual object.

In a further embodiment a method for determining imaging parameters for visualization considers the brightness of a virtual object and determines an imaging parameter which sets a longer exposure time if the brightness of the virtual object is low and a shorter exposure time if the brightness of the virtual object is high.

In a further embodiment a method for determining imaging parameters for visualization considers the brightness of a virtual object and determines an imaging parameter which sets a shorter exposure time if the brightness of the virtual object is low and a longer exposure time if the brightness of the virtual object is high.

In a further embodiment a method for determining imaging parameters for visualization considers the brightness of a virtual object and the distance of the virtual object to a camera and determines an imaging parameter which sets a shorter exposure time if the brightness of the virtual object is high and/or if the distance between the virtual object and the camera is short and/or which sets a longer exposure time if the brightness of the virtual object is low and/or if the distance between the virtual object and the camera is large.

In a further embodiment a method for determining imaging parameters for pose determination considers the distance between a part of the real environment and the camera and determines an imaging parameter which sets the focus distance to the distance of the part of the real environment.

In a further embodiment a method for determining imaging parameters for pose determination considers the brightness of a part of the real environment and determines an imaging parameter which sets a shorter exposure time if the brightness of the virtual object is high and a longer exposure time if the brightness of the virtual object is low.

In an embodiment determining imaging parameters based on poses and virtual object properties considers the brightness of a virtual object (which is a virtual object property) and the distance of the virtual object to a camera (which is derived from poses) and determines an imaging parameter which sets a shorter exposure time if the brightness of the virtual object is high and if the distance between the virtual object and the camera is short and which sets a longer exposure time if the brightness of the virtual object is low and if the distance between the virtual object and the camera is large.

In a further embodiment determining imaging parameters based on poses and virtual object properties may determine imaging properties indicative of an exposure time if the virtual object properties are indicative of the brightness of the virtual object and it further may determine imaging properties indicative of a focal plane if the virtual object properties are indicative of that the virtual object always shall be in focus and it further may determine imaging properties indicative of zoom settings if the virtual object properties are indicative of a range of size that the virtual object should have in the augmented image and it further may determine imaging properties indicative of bit depth if the virtual object properties are indicative of dynamic range.

Referring again to the Figures, the RWAA (101), (901), (1001), (1051) or (1071) may be implemented as a camera system or as a part of a camera system. In another embodiment, a part of the RWAA (101), (901), (1001), (1051) or (1071) may be implemented in a camera system.

All of the components disclosed in FIGS. 1, 9, 10a, 10b, or 10c may be completely implemented by a system comprising at least one camera device and at least one processing device. The at least one camera device may comprise at least one of light field cameras and non-light field cameras (also called conventional cameras). The at least one processing device may comprise at least one of a mobile device (e.g. mobile phone, tablet), desktop computer, and server computer. The at least one camera device and the at least one processing device may communicate via cable or wirelessly.

Any step or method disclosed herein may be performed by one or more processing units of at least one camera and/or at least one processing device.

In an embodiment, the components disclosed in FIG. 1, 9, 10a, 10b, or 10c may be implemented in a mobile device equipped with a camera (e.g. a mobile phone or a tablet). In another embodiment, the components disclosed in FIG. 1, 9, 10a, 10b, or 10c may be implemented in a light field camera device. In a further embodiment, the components disclosed in FIG. 1, 9, 10a, 10b, or 10c may be implemented in a light field camera device connected to a computer device, e.g. a desktop computer, a mobile computer, or a server computer.

Generally, in the following, a further explanation of terms is given and the following further aspects and embodiments may be applied in connection with aspects of the invention.

Camera:

A camera is often called an imaging device or capturing device. The proposed invention can generally be applied with any camera capable of providing images. It is not restricted to cameras providing color images in the RGB format. It can also be applied to any other color format and also to monochrome images, for example to cameras providing images in grayscale format or YUV format.

A camera often has intrinsic parameters including focal length and principal point. A camera may capture light that is visible to the human eye. A camera may also capture light that is invisible to the human eye, such as infrared light. For example, the camera may be a thermal imaging camera or infrared camera.

A camera providing an image with depth data is often called a depth camera. A depth camera system could be a time of flight (TOF) camera system or a passive stereo camera or an active stereo camera based on structured light. A depth camera may provide images whose pixels represent only depth data. A depth camera may capture both the light and depth data in a single image, e.g. RGB-D image. For the depth camera, the depth data does not need to be provided in the same resolution as the (color/grayscale) image.

For a standard camera model with a simplified pinhole or fisheye camera model, only light falling through a single point, i.e. the pinhole, is measured.

This invention may further use a light field camera that could capture an image from multiple different viewpoints and optionally with different focuses for the different viewpoints. Light field cameras are not limited to capturing only light falling through a single point, but measure also a plurality of light intensities incident at different locations. A non-light field camera is called conventional camera herein.

Image:

An image is any data depicting or recording visual information or perception. The image could be a 2-dimensional image. The image could also be a 3-dimensional image, e.g. a depth image.

An image could be a real image or a synthetic image. A real image may be captured by a camera. For example, the camera could capture an object of interest or a part of the object of interest in one real image. A synthetic image may be generated automatically by a computer or manually by a human. For example, a computer rendering program (e.g. based on OpenGL) may generate a synthetic image of an object of interest or a part of the object of interest. A synthetic image may be generated from a perspective projection as it is captured by a camera. A synthetic image may be generated according to orthogonal projection.

An image may capture an object that reflects, refracts, and/or emits light that is visible and/or invisible to human eyes. The image may be in the RGB format. It can also be applied to any other color format and also to monochrome images for example in grayscale format or YUV format. For example, an infrared image could capture an object that reflects, refracts, and/or emits light that is invisible to the human eyes.

A depth image may be a 2D (color/grayscale) image with a corresponding depth map. The depth images do not need to be provided in the same resolution as the 2D image. The depth image may also be called 3D image. A depth image may only provide depth data.

Feature:

A feature of an object is used to denote a piece of information related to the object. The piece of information may be visually perceivable to anatomical eyes or optical imaging devices. For example, a real object may emit or reflect visible light that could be captured by human eyes or cameras. The real object may also emit or reflect invisible light that could not be captured by human eyes, but could be captured by a camera (i.e. is optically perceivable). In another example, a feature may be associated with a virtual object (i.e. computer-generated object). A feature of the virtual object may be known or detectable in a computer or computer program, like computer graphic simulation software.

A feature may describe specific colors and/or structures, such as blobs, edges points, a particular region, and/or more complex structures of the real object. A feature may be represented by an image patch (e.g. pixel intensity) or a high level descriptor (e.g. SIFT, SURF).

A feature may have 3D position and/or 3D orientation information in 3D Euclidean space relative to a coordinate system of the real object. This is often called a 3D feature.

A feature may also be expressed in 2D space. For example, a feature (i.e. the piece of information related to the real object) may be extracted from an image of the real object captured by a camera, and thus the feature may have 2D image position and/or orientation in a coordinate system of the image. This is often called 2D feature or image feature. When a camera could provide depth information, a feature extracted from an image of the camera may also have 3D position and/or orientation information.

A feature could be described by an equation that describes a geometric shape, for example a point, a ray, a straight line, a circle, a cone, or a cylinder.

Methods to detect features in an image that could be used in a method of the invention include but are not limited to local extrema of Laplacian of Gaussian (LoG), Difference of Gaussians (DoG) or Determinant of Hessian (DoH), Maximally stable extremal regions (MSER), Harris features, or learning-based corner detectors such as FAST. Also methods that detect edges (edge elements) are suitable to be used in such method. The feature detection method to be used is not limited to approaches working on a 2D intensity grayscale image, but can also be performed on images in any other color space, including RGB, HSV, and Lab, or range images that either exclusively contain depth information or provide both depth and intensity information.

Pose:

A pose of a first object relative to a second object (e.g. a coordinate system associated with the second object) describes a rigid transformation including a translation and/or a rotation between the first object and the second object. A pose of the second object relative to the first object may be considered equivalent to a pose of the first object relative to the second object, as they describe the same spatial relationship.

In 3D space, a pose may include information for six degrees of freedom (DOFs) or a part of the six DOFs. The six DOFs include three DOFs for translations and three DOFs for rotations. In 2D space, a pose may include information for three degrees of freedom (DOFs) or a part of the three DOFs. The three DOFs include two DOFs for translations and one DOF for rotations.

A motion of a real object (e.g. a camera) describes a spatial relationship or a rigid transformation between the real object at one position and the real object at another position in a common coordinate system.

Camera Pose Determination:

A pose of a camera relative to a real world (e.g. a real object or a real environment) may be determined based on an image of at least part of the environment captured by the camera.

In one implementation, a model of the real object can be used for model based matching. The model based matching could for example be based on point features, edge features, or image patches of any size and form. While point features are frequently used for highly textured objects, edge features are preferred if the real object has little texture. Model based matching requires the image used for pose determination to contain at least part of the real object described by the model. Note that the real object could for example also include a fiducial marker in the environment. A model of the real object could be described by points, edges, polygons or their combinations. A model of the real object may also contain texture information, e.g. colors.

Determining a pose of the camera can also be realized by using a visual marker. This requires the visual marker at a known position relative to the real object. In this case, a camera pose with respect to the real object could be determined according to a camera pose with respect to the visual marker, which is estimated based on an image of the camera containing the visual marker. It is not necessary for the image to contain at least part of the real object when the visual marker is used for the camera pose determination.

A camera motion between two camera positions may be determined by camera poses at the two camera positions. A camera motion may also be determined according to two images captured by the camera at the two camera positions respectively. For example, common image features in the two images may be employed to determine the camera motion.

A task of camera pose estimation is to compute a spatial relationship or a transformation between a camera and a reference object (or environment). Camera motion estimation is to compute a spatial relationship or a transformation between a camera at one position and the camera at another position. Camera motion may be also known as camera pose which describes a pose of a camera at one position relative to the same camera at another position. Camera pose or motion estimation is also known as tracking a camera. A spatial relationship or transformation describes a translation, a rotation, or their combination in 3D space.

Object Model:

An object model describes a geometry and/or texture of an object or a generic geometry and/or texture of a group of objects. An object model may be also called a digital model as it is digitally represented or stored.

Geometry refers to one or more attributes of an object including, but not limited to, shape, form, surface, symmetry, geometrical size, dimensions, and/or structure. A dimension of a geometry may be 2D (e.g. a 2D plane), 3D (e.g. a 3D cube), or 4D (3D in space+1D in time). For example, a model may include changes of geometry and/or texture of the object with respect to the time.

An object model of a real object or a virtual object could be represented by a CAD model, a polygon model, a point cloud, a volumetric dataset, and/or an edge model, or use any other representation.

At least part of a geometry may be mathematically represented by at least one coordinate (i.e. discrete representation) or by at least one mathematic formula (i.e. continuous representation) in a 2D or 3D coordinate system. For example, a circle or a sphere may be represented by a set of points or a mesh or a set of edges (discrete representation) or represented by an equation in a 2D or 3D coordinate system. A circle that is a 2D shape may be defined in a 2D or 3D space. A sphere that is a 3D geometry may be defined in a 2D space as a projection of the sphere (i.e. 3D shape) onto the 2D space.

A 3D model can further, for example, be represented as a model comprising 3D vertices and polygonal faces and/or edges spanned by these vertices. Edges and faces of the model may also be represented as splines or NURBS surfaces. A 3D model may be accompanied by a bitmap file describing its texture and material where every vertex in the polygon model has a texture coordinate describing where in the bitmap texture the material for this vertex is stored. A 3D model can also be represented by a set of 3D points as, for example, captured with a laser scanner. The points might carry additional information on their color, intensity and/or any other additional information related to particular point.

A model for an object may further include a material of the object. A material of the object could be represented by textures and/or colors in the model and could, for example, represent the reflection or radiance transfer properties of the model. A model of an object may use different representations for different parts of the object.

A 2D model of an object may describe 2D geometry of at least part of the object. A 3D model of an object may describe 3D geometry of at least part of the object. The object may be a real object or a virtual object. A generic model may describe generic geometry and/or texture of a group of objects. For example, a 3D model may be specific for an object. A generic 3D model may not be specific for an object, but may describe a generic geometry for a group of similar objects. The similar objects may belong to a same object type and share some common properties. For example, faces of different people are belonging to a same object type having eye, mouth, ear, nose, etc. Cars of different designs or from different brands are belonging to a same object type having four tires, at least two doors, and a front window glass, etc.

A generic 3D face model may not be the same as any real existing individual face, but it may be similar to an existing individual face. For example, the silhouette of the face of the 3D model may not exactly match the silhouette of the existing individual face, but they are all the shape of eclipse. A generic 3D face model may be determined according to a plurality of 3D models of faces. The face model can also be already further processed, for example containing the principal component analysis with mean and variances for at least one of face specific characteristics.

An object model may include a kinematic model for an articulated object, like joint parameters for the kinematic chain. An object model may include parameters indicative of deformable properties for a deformable object. An object may also be rigid.