In many applications of machine learning, the machine learning model accuracy is the most important consideration, and knowing the uncertainty of those predictions is not critical. For example, for a clothing recommendation engine, it is important that on average it suggests clothes that a customer would like to buy. It is acceptable for it to occasionally recommend an article of clothing that a customer dislikes, as long as its average performance is high.
At Citrine, we recognize that building accurate models for materials properties is not enough.
In order for data-driven models to be useful in materials science applications, it is critical to have a reliable estimate of model uncertainty reported with every prediction.
For example, say that we have trained a model to predict band gap based on the Strehlow and Cook experimental dataset. We want to make predictions for the band gaps of a couple new compounds, tin monoxide (SnO) and nickel oxide (NiO). Our model predicts values of 2.4 eV and 2.8 eV respectively. The key question is, “How confident is our model in these predictions?”
There are many different sources of uncertainty in data-driven models. If the model was fit to noisy training data, then that noise will cause uncertainty in the model. If the model is fit to only a small number of data points, it will also have higher uncertainty. Another 
important source of uncertainty is extrapolation. For example, if we trained a model on the blue dots in the figure to the right, then tried to make a prediction at the red X, our prediction would have high uncertainty. Similarly, data-driven models are unreliable at making predictions on materials that are significantly different from any of the materials in the training set.
At Citrine, all our predictions come with uncertainty estimates. We have developed, implemented, and validated cutting edge uncertainty quantification methods for data-driven materials models. For more details on our uncertainty quantification techniques and how they can be used to accelerate materials design, please see our recent paper.1
In the cases of SnO and NiO, our predictions are shown below.
These plots show the probability distribution function for our prediction. For example, in the case of SnO, the mean value of the distribution is 2.45 eV and the uncertainty of 0.78 eV is based on the spread of the distribution at one standard deviation. Since the uncertainty estimates are based on the standard deviation of the distribution, they are a 68% confidence interval, i.e. the probability that the true value is within 0.75 eV of the prediction (2.45 eV) is 68%.
The model uncertainty for NiO (1.41 eV) is much higher than for SnO (0.78 eV), in part because the training set included far fewer compounds containing nickel than tin. The higher uncertainty in the NiO predictions reflects the fact that the model is extrapolating at this point. The true band gap for SnO is approximately 2.5 eV and for NiO is approximately 3.8 eV.2
At Citrine, we know that uncertainty estimates are critical for assessing model confidence when using data-driven models for real engineering applications. We are proud to be leading the field by providing well-calibrated uncertainty estimates for all our predictions.3
- Ling, Julia, et al. “High-Dimensional Materials and Process Optimization using Data-driven Experimental Design with Well-Calibrated Uncertainty Estimates.” Integrating Materials and Manufacturing Innovation (2017).
- Wong, Terence KS, et al. “Current status and future prospects of copper oxide heterojunction solar cells.” Materials 9.4 (2016): 271.
- This work was funded in part by Argonne National Laboratories through contract 6F-31341, associated with the R2R Manufacturing Consortium funded by the Department of Energy Advanced Manufacturing Office.
For a materials science student with a background in computer science, finding somewhere to combine work in these two fields is already an uncommon opportunity. Applying machine learning and data mining techniques to answer questions in materials science has been an opportunity that is really unique to Citrine. The projects that I’ve worked on have paired materials intuition with an understanding of data science and machine learning to build models for various material properties. This goes hand in hand with learning how to visualize and communicate machine learning results to people in the materials science field who can be unfamiliar with machine learning concepts. In addition to various modeling projects, I’ve also been able to contribute to the continued development of our in-house machine learning infrastructure. Working as an intern in a small, fast-moving team has been especially rewarding as projects I’ve worked on actually see the light of day. Work that I’ve done has been delivered to customers, and demo’d to investors and potential customers. It has been an honor and a rush to be able to work at Citrine as we try to fundamentally change how data is used in the field of materials science.