While photometric stereo has some very interesting use cases and strengths, it also comes with limitations that many people are not aware of, so I decided to sum up the basics in this post.
Error Sources
Calibrated photometric stereo requires several photographs of a surface under different, known lighting conditions – the more, the better. Known lighting conditions in this case ideally means that for each pixel and light combination you know the light vector and light intensity – for point lights this translates to knowing the exact 3D position of each light and the 2D position on the plane of each pixel, as well as having measured the spatial radiation pattern of each light source. Of course all error sources inherent to taking photographs are also relevant for photometric stereo.
Parameters of a BRDF of your choice can be fitted to the captured data, and in most cases Lambertian reflectance is assumed. Even if you use more sophisticated BRDFs, they are nevertheless a somewhat poor match for the material to be scanned in most cases. On top of that, shadows and interreflections are very difficult to account for and many solvers don’t even attempt to.
The direct output of a photometric stereo solver will be a normal vector for each pixel along with the rest of the BRDF parameters describing the appearance. A height map can be obtained via integration of the normal map causing any errors in the normal vectors to accumulate.
Quantifying Accuracy
The accuracy of a photometric stereo scanner is very difficult to quantify because of the complex error sources and thus highly case by case dependent performance characteristics. In literature, an ideal case of using a Spectralon sphere is often used for benchmarking and the angular error can be as low as 2° for the estimated normal vector. The accuracy will be significantly worse for real world use cases.
However, the fact that photometric stereo measures the normal vector instead of the 3D geometry is a very unique property. This means that it can pick up small scratches with a depth of a few micrometres where other methods may fail, even though the height map could be off by centimetres in some places. As such, photometric stereo has some interesting use cases in surface inspection, cultural heritage or even forensics.
If any kind of accuracy for the height map along with the advantages of photometric stereo is necessary, then combining it with other methods like structured light or photogrammetry is the way to go.
Material Types
Generally, materials can be split into two types in terms of their appearance: dielectrics and metallics. Dielectrics have both a diffuse and specular reflectance, metals only have a specular reflectance. When building a photometric stereo setup you will need to decide which types of reflections you want to target.
- Diffuse reflection has the advantage that it can be approximated by the Lambertian reflectance with the disadvantage that subsurface scattering (SSS) may blur normal maps obtained by photometric stereo.
- Specular reflection is not affected by subsurface scattering but its shape will depend on the surface roughness of the material. The lower the roughness, the smaller the specular lobe and the more images need to be obtained by the setup.
Many scanners only target the diffuse reflection because it simplifies the approach significantly. Scanners that also target the specular reflection either capture normal images and jointly solve for the total reflection or capture cross / parallel polarized images which allow for separation of the reflection types.
The performance of each photometric stereo setup will also depend heavily on the type of materials it should scan. Here are a few examples in the context of appearance capture:
- Leaves: Easily doable with simple setups, maybe some blurring due to SSS.
- Fabrics: Easily doable with simple setups, but the materials will most likely need manual tweaking to look good.
- Skin: Expect blurring of the normals with a cross polarized setup due to SSS, setups targeting the specular reflection of skin are fairly well researched, but also significantly more complex.
- Rough metals (e.g. old coins): Due to the absence of the diffuse reflection, a significant amount of light sources is probably needed, but it’s doable.
- Shiny metals (e.g. jewelry): You’re probably out of luck. Though, I do know of one research project capable of this: SpecScanning (one of the many great projects by the USC Institute for Creative Technologies)
If you’re looking to apply photometric stereo to a different use case such as defect detection, then the feasibility will depend on your exact use case and even shiny metals could be doable in some cases.
Software
There aren’t a lot of software packages available for photometric stereo. If you’re looking into doing something even slightly advanced like proper calibration of the device or solving for BRDFs other than Lambertian, then you will almost certainly need custom software. There are a few open source projects you can use as a reference though.
FAQ
- How many images under different illumination do I need for photometric stereo?
Mathematically, at least 3. Realistically, at least 8. More complex setups use up to a few hundred. - Can I use photometric stereo by itself for getting an accurate 3D model / height map of a surface/object?
No. - Can I use photometric stereo for detecting small defects in a surface?
Probably. - What happens when the scanner is not calibrated correctly?
The resulting textures will likely have large gradients. - How difficult is getting an accurate roughness map using photometric stereo?
Difficult. - Can photometric stereo handle discontinuities in the height map?
There may be some research projects attempting this, but for any real world use case the answer is no. - How well does photometric stereo handle steep angles?
Not well.