Sony Patent | Information Processing Apparatus, Information Processing Method, Program, And Cell Observation System
Publication Number: 20200125030
Publication Date: 20200423
Applicants: Sony
Abstract
An information processing apparatus according to an embodiment of the present technology includes: a calculation unit; and an amplitude replacement unit. The calculation unit repeatedly executes first light propagation calculation for propagating, from a sensor surface of an image sensor to a support surface that supports a cell to be observed, a first complex amplitude distribution that includes a light intensity distribution of a hologram of the cell obtained on the sensor surface, and second light propagation calculation for propagating, from the support surface to the sensor surface, a second complex amplitude distribution obtained as a result of the first light propagation calculation. The amplitude replacement unit replaces, in the second light propagation calculation, an amplitude component of the second complex amplitude distribution with a predetermined representative amplitude value at least once.
TECHNICAL FIELD
[0001] The present technology relates to an information processing apparatus, an information processing method, a program, and a cell observation system that are capable of reconstructing an image of a cell from a hologram.
BACKGROUND ART
[0002] A phase-contrast microscope that is generally used as a microscope for observing a cell needs Koehler illumination for illumination and a magnifying optical system for observation, which makes the system large, and costs a lot. For this reason, in recent years, a lensless microscope including only a light source and a general image sensor has attracted attention.
[0003] The lensless microscope has an in-line hologram as a basic principle, and is capable of reconstructing an image of an object from an imaged hologram by calculation. However, in such an in-line hologram, since the above-mentioned image sensor is capable of recording only information (square value of the amplitude) regarding a light intensity, it is necessary to recover information regarding the light phase in order to acquire a reconstructed image of the object.
[0004] As a method of recovering information regarding the hologram phase, an iterative phase retrieval method in which the phase information is recovered by repeating propagation with a plurality of holograms imaged at different wavelengths as restraint conditions has been reported (e.g., Non-Patent Literature 1).
CITATION LIST
Non-Patent Literature
[0005] Non-Patent Literature 1: A. Lambrechts, “Lens-free digital in-line holographic imaging for wide field-of-view, high resolution and real-time monitoring of complex microscopic objects”, Proc. of SPIE, Vol. 8947, 2014
DISCLOSURE OF INVENTION
Technical Problem
[0006] However, even with the technology described in Non-Patent Literature 1, there is a problem that the information regarding the hologram phase cannot be sufficiently recovered and artifacts of the reconstructed image cannot be completely removed.
[0007] In view of the circumstances as described above, it is an object of the present technology to provide an information processing apparatus, an information processing method, a program, and a cell observation system that are capable of correctly recovering information regarding a hologram phase and reducing artifacts of a reconstructed image.
Solution to Problem
[0008] In order to achieve the above-mentioned object, an information processing apparatus according to an embodiment of the present technology includes: a calculation unit; and an amplitude replacement unit.
[0009] The calculation unit repeatedly executes first light propagation calculation for propagating, from a sensor surface of an image sensor to a support surface that supports a cell to be observed, a first complex amplitude distribution that includes a light intensity distribution of a hologram of the cell obtained on the sensor surface, and second light propagation calculation for propagating, from the support surface to the sensor surface, a second complex amplitude distribution obtained as a result of the first light propagation calculation.
[0010] The amplitude replacement unit replaces, in the second light propagation calculation, an amplitude component of the second complex amplitude distribution with a predetermined representative amplitude value at least once.
[0011] As a result, the phase component of the complex amplitude distribution of the hologram is appropriately updated, and it is possible to acquire a reconstructed image of the cell in which the phase component has been sufficiently retrieved. That is, it is possible to reconstruct the sample surface from the defocused hologram.
[0012] The amplitude replacement unit may replace an amplitude component of the first complex amplitude distribution with an amplitude component of a different hologram acquired under a different imaging condition every time the first light propagation calculation is executed.
[0013] As a result, the frequency of restraining the amplitude component of the first complex amplitude distribution increases, and the number of times the propagation calculation necessary for phase retrieval is executed is reduced.
[0014] The different hologram may be one of a plurality of holograms having different wavelengths of the illumination light.
[0015] The different hologram may be one of a plurality of holograms having different distances from the support surface.
[0016] The amplitude replacement unit may replace the amplitude component of the second complex amplitude distribution with the predetermined representative amplitude value every time the second light propagation calculation is executed.
[0017] The predetermined representative amplitude value may be an average value of complex amplitude distributions obtained as results of the first light propagation calculation.
[0018] As a result, the amplitude component of the complex amplitude distribution of the hologram is smoothed, and the calculation load is reduced.
[0019] The predetermined representative amplitude value may include a value obtained by multiplying the average value by a predetermined correction coefficient, and the amplitude replacement unit may cause the correction coefficient to differ for each pixel region.
[0020] As a result, the frequency of restraining the amplitude component of the second complex amplitude distribution is adjusted.
[0021] The predetermined representative amplitude value may include a value obtained by multiplying the average value by a predetermined correction coefficient, and the amplitude replacement unit may cause the correction coefficient to differ every time the second light propagation calculation is executed.
[0022] As a result, the frequency of restraining the amplitude component of the second complex amplitude distribution is adjusted.
[0023] In order to achieve the above-mentioned object, an information processing method according to an embodiment of the present technology includes:
[0024] repeatedly executing first light propagation calculation for propagating, from a sensor surface of an image sensor to a support surface that supports a cell to be observed, a first complex amplitude distribution that includes a light intensity distribution of a hologram of the cell obtained on the sensor surface, and second light propagation calculation for propagating, from the support surface to the sensor surface, a second complex amplitude distribution obtained as a result of the first light propagation calculation.
[0025] In the second light propagation calculation, an amplitude component of the second complex amplitude distribution is replaced with a predetermined representative amplitude value at least once.
[0026] In order to achieve the above-mentioned object, a program according to an embodiment of the present technology causes an information processing apparatus to execute the steps of:
[0027] repeatedly executing first light propagation calculation for propagating, from a sensor surface of an image sensor to a support surface that supports a cell to be observed, a first complex amplitude distribution that includes a light intensity distribution of a hologram of the cell obtained on the sensor surface, and second light propagation calculation for propagating, from the support surface to the sensor surface, a second complex amplitude distribution obtained as a result of the first light propagation calculation; and replacing, in the second light propagation calculation, an amplitude component of the second complex amplitude distribution with a predetermined representative amplitude value at least once.
[0028] In order to achieve the above-mentioned object, a cell observation system according to an embodiment of the present technology includes: a light source; a sample holder; an image sensor; and a reconfiguration processing unit.
[0029] The light source emits illumination light.
[0030] The sample holder has a support surface that supports a cell to be observed.
[0031] The image sensor has a sensor surface that receives a hologram generated by interference between transmitted light and diffracted light, the illumination light being separated by the cell into the transmitted light and the diffracted light.
[0032] The reconfiguration processing unit repeatedly executes first light propagation calculation for propagating, from the sensor surface to the support surface, a first complex amplitude distribution that includes a light intensity distribution of the hologram obtained on the sensor surface, and second light propagation calculation for propagating, from the support surface to the sensor surface, a second complex amplitude distribution obtained as a result of the first light propagation calculation, and replaces, in the second light propagation calculation, an amplitude component of the second complex amplitude distribution with a predetermined representative amplitude value at least once.
BRIEF DESCRIPTION OF DRAWINGS
[0033] FIG. 1 is a schematic diagram showing a configuration example of a cell observation system according to a first embodiment of the present technology.
[0034] FIG. 2 is a flowchart showing an information processing method for an information processing apparatus according to the above-mentioned embodiment.
[0035] FIG. 3 is a block diagram showing a procedure until the cell observation system according to the above-mentioned embodiment acquires a reconstructed image of a cell.
[0036] FIG. 4 is a block diagram showing a procedure of pre-processing by a pre-processing unit in the above-mentioned embodiment.
[0037] FIG. 5 is a diagram showing calculation processing (algorithm) in iteration in an iterative phase retrieval method executed by a reconfiguration processing unit in the above-mentioned embodiment.
[0038] FIG. 6 is a block diagram showing a procedure of amplitude replacement processing by an amplitude replacement unit in the above-mentioned embodiment.
[0039] FIG. 7 is a block diagram showing a procedure of amplitude replacement processing by the amplitude replacement unit in the above-mentioned embodiment.
[0040] FIG. 8 is a diagram comparing the calculation results of an existing method and the iterative phase retrieval method in the above-mentioned embodiment.
[0041] FIG. 9 is a diagram comparing the calculation results of an existing method and the iterative phase retrieval method in the above-mentioned embodiment.
[0042] FIG. 10 is a diagram showing a reconstructed image of a cell acquired by the iterative phase retrieval method in the above-mentioned embodiment together with an image of a cell captured by a quantitative phase microscope.
[0043] FIG. 11 is a graph showing the phase values of the above-mentioned various images.
[0044] FIG. 12 is a diagram showing calculation processing (algorithm) in iteration in the iterative phase retrieval method executed by a reconfiguration processing unit in a second embodiment of the present technology.
[0045] FIG. 13 is a diagram showing calculation processing (algorithm) in iteration in the iterative phase retrieval method executed by a reconfiguration processing unit in a third embodiment of the present technology.
[0046] FIG. 14 is a block diagram showing a procedure of amplitude replacement processing by an amplitude replacement unit in a modified example of the present technology.
MODE(S)* FOR CARRYING OUT THE INVENTION*
[0047] Hereinafter, embodiments of the present technology will be described with reference to the drawings.
First Embodiment
[0048] FIG. 1 is a schematic diagram showing a configuration example of a cell observation system 100 according to a first embodiment of the present technology.
[0049] As shown in FIG. 1, the cell observation system 100 includes a light source 10, an observation stage 20, an image sensor 30, a sensor/light source control unit 40, an input unit 50, and an information processing apparatus 60. Note that in FIG. 1, an X axis, a Y axis, and a Z axis orthogonal to each other are show.
[0050] The light source 10 is configured to be capable of applying, for example, illumination light having wavelengths (.lamda..sub.R: 636 nm, .lamda..sub.G: 515 nm, .lamda..sub.B: 470 nm) corresponding to RGB to a cell C on the observation stage 20.
[0051] In the case where the illumination light from the light source 10 is applied to the cell C (observation target), this illumination light is divided into transmitted light and diffracted light. The transmitted light interferes with the diffracted light on the image sensor 30, thereby generating a hologram on the image sensor 30. The transmitted light can be referred to also as reference light for generating a hologram. This hologram (interference fringe) can be calculated on the basis of the Fresnel-Kirchhoff diffraction formula or Rayleigh-Sommerfeld diffraction formula (see formula (1)) described below.
[0052] The light source 10 in this embodiment is typically a partially coherent LED light source, but may be configured to increase temporal coherence by a band pass filter and spatial coherence by a pinhole.
[0053] The observation stage 20 supports a sample holder H that supports the cell C. The sample holder H has a support surface S1 that supports the cell C to be observed. The sample holder H is not particularly limited. However, the sample holder H is typically a preparation including a slide glass and a cover glass and has a light transmission property.
[0054] The observation stage 20 may be configured to be movable in the Z-axis direction. As a result, a distance Z between the support surface S1 and an image sensor surface S2 described below is adjusted, and the position of the image sensor 30 relative to the cell C can be adjusted.
[0055] The observation stage 20 has an area having a light transmission property, which causes the illumination light of the light source 10 to be transmitted therethrough, and the sample holder H is installed on this area. The area having a transmission property provided on the observation stage 20 may be formed of glass or the like, and may include an opening that communicates the upper and lower surfaces of the observation stage 20 in the Z-axis direction.
[0056] Note that although the cell C is employed as an object of the cell observation system 100 in this embodiment, the present technology is not limited thereto. For example, all of those derived from living bodies, such as a tissue, a sperm, a fertilized egg, a microorganism, may be employed as the object.
[0057] The image sensor 30 records the hologram of the cell C generated on the image sensor surface S2, and outputs image data regarding this hologram to the information processing apparatus 60. The image sensor 30 is, for example, a general image sensor such as a CCD sensor and a CMOS sensor. For this reason, in the recorded hologram on the image sensor surface S2, only a light intensity distribution (square value of amplitude) is recorded. Note that the image sensor surface S2 is a light reception surface that receives the hologram of the cell C.
[0058] The sensor/light source control unit 40 is connected to the light source 10 and the image sensor 30 wirelessly or by wire, and configured to be capable of controlling them. The sensor/light source control unit 40 controls the light source 10, and thus, the wavelength of illumination light to be applied to the cell C is switched, for example.
[0059] The input unit 50 is an operation device that inputs, to the information processing apparatus 60, operation information by a user. The input unit 50 may be an operation device such as a keyboard and a mouse, a touch panel, or the like.
[0060] The information processing apparatus 60 includes hardware necessary for a computer, such as a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive). The CPU loads the program according to the present technology stored in the ROM or HDD into the RAM and executes it, thereby executing an iterative phase retrieval method for the information processing apparatus 60 described below.
[0061] The program is installed in the information processing apparatus 60 via, for example, various storage media (internal memory). Alternatively, the program may be installed via the Internet or the like. In this embodiment, as the information processing apparatus 60, for example, a PC (Personal Computer) or the like is used. However, another arbitrary computer may be used.
[0062] [Information Processing Apparatus]
[0063] The information processing apparatus 60 includes an image acquisition unit 61, a pre-processing unit 62, a reconfiguration processing unit 63, and a display control unit 64.
[0064] The image acquisition unit 61 acquires, from the image sensor surface S2 on which the image data of the plurality of holograms in which the cell C has been imaged under different conditions is recorded, the image data.
[0065] For example, in the case where the light source 10 individually applies illumination light having the wavelengths .lamda..sub.R, .lamda..sub.G, and .lamda..sub.B to the cell C, image data regarding holograms g.sub..lamda.R, g.sub..lamda.G, and g.sub..lamda.B corresponding to the wavelengths is acquired.
[0066] Alternatively, in the case where the light source 10 applies illumination light having a predetermined wavelength A to the cell C, image data regarding various holograms g.sub.Z1, g.sub.Z2, and g.sub.Z3 recorded on the image sensor surface S2 at first to third positions Z1, Z2, and Z3, respectively, with respect to the cell C is acquired.
[0067] The pre-processing unit 62 performs various types of correction on the image data regarding the hologram output from the image acquisition unit 61 so that iterative processing in an iterative phase retrieval method described below is appropriately performed.
[0068] The reconfiguration processing unit 63 includes a calculation unit 63a and an amplitude replacement unit 63b. The reconfiguration processing unit 63 retrieves the phase component of the complex amplitude distribution relating to the hologram lost on the image sensor surface S2 by repeating propagation between the image sensor surface S2 and the support surface S1 with the hologram output from the pre-processing unit 62 as the restraint condition.
[0069] Specifically, the amplitude replacement unit 63b repeats replacement of the amplitude component while transitioning the hologram by the light wave propagation calculation by the calculation unit 63a, and thus the lost phase component is retrieved. At this time, the reconfiguration processing unit 63 repeatedly executes a cycle for replacing the amplitude component of the complex amplitude distribution of the hologram obtained from the result of the propagation calculation with the actually measured amplitude component so that only the phase component remains.
[0070] Here, the “propagation of the hologram” in this embodiment means executing light wave propagation calculation for calculating the complex amplitude distribution (g(x,y,0)) in the hologram of the propagation destination on the basis of the Rayleigh-Sommerfeld diffraction integral represented by the following formula (1) from the complex amplitude distribution (g(x,y,z)) in the hologram of the propagation source.
[ Math . 1 ] g ( x , y , 0 ) = .intg. .intg. g ( x ’ , y ’ , z ’ ) exp ( i 2 .pi. r ’ .lamda. - 1 ) r ’ - z r ’ ( 1 / 2 .pi. r ’ + 1 / i .lamda. ) dxdy r ’ = ( ( x - x ’ ) 2 + ( y - y ’ ) 2 + z 2 ) 1 / 2 k = 2 .pi. / .lamda. ( 1 ) ##EQU00001##
[0071] Since the calculation takes time in the state of the integral form of the formula (1), the following formula (2) obtained by converting the formula (1) into a product form of Fourier transform is adopted in this embodiment. Note that in the formula (2), G represents the Fourier transform of g, and u and v represent spatial frequency components in the X direction and the Y direction.
[ Math . 2 ] g ( x , y , 0 ) = F - 1 ( G ( u , v , z ) exp ( - i 2 .pi. w ( u , v ) z ) ) w ( u , v ) = { ( .lamda. - 2 - u 2 - v 2 ) 1 / 2 u 2 + v 2 .ltoreq. .lamda. - 2 0 otherwise ( 2 ) ##EQU00002##
[0072] As will be described below, the reconfiguration processing unit 63 in this embodiment recalculates, from the complex amplitude distribution of the hologram propagated from the image sensor surface S2 to the support surface Slat a predetermined wavelength, the complex amplitude distribution of the hologram to be propagated from the support surface S1 to the image sensor surface S2 at a wavelength different from the above-mentioned wavelength. Therefore, in this embodiment, a calculation formula in which the formula (2) is replaced with the following formula (3) is adopted.
[Math. 3]
g.sub..lamda.G(x,y,z)=F.sup.-1{G.sub..lamda.B(u,v,z)exp[i2.pi.(w.sub.G(u- ,v)-w.sub.B(u,v))z]} (3)
[0073] The formula (3) means that the complex amplitude distribution of the hologram g.sub..lamda.G to be propagated from the support surface S1 to the image sensor surface S2 at the wavelength .lamda..sub.G is calculated from the complex amplitude distribution of the hologram g.sub..lamda.B propagated from the image sensor surface S2 to the support surface S1 at the wavelength .lamda..sub.B.
[0074] In this embodiment, the calculation unit 63a repeatedly executes light wave propagation calculation between the sensor surface S2 and the support surface S1 on the basis of the formulae for calculating propagation, i.e., the formulae (2) and (3).
[0075] For example, in the case where the amplitude replacement unit 63b does not execute amplitude replacement in the support surface S1, the propagation calculation based on the formula (3) is executed. Meanwhile, in the case of executing amplitude replacement, the complex amplitude distribution of the hologram g.sub..lamda.G to be propagated from the support surface S1 to the image sensor surface S2 at the wavelength .lamda..sub.G is calculated on the basis of the formula (2) after replacing the amplitude component of the complex amplitude distribution of the hologram g.sub..lamda.B propagated from the image sensor surface S2 to the support surface S1 at the wavelength .lamda..sub.B with a predetermined representative amplitude value.
