Why “Synthetic Data” Costs More
“Synthetic data” has been in technically sound use since the 1990s, but it has recently become a hot topic due to the mistaken belief that it offers cost savings for brute force AI models that will never get enough training data.
This is a misconception:
- Nothing changes the fact that useful data is now, and has always been, expensive to obtain.
- In its traditional uses, synthetic data is more expensive than real data because it requires additional work to provide some rigorous benefit over real data.
- In its newer uses, synthetic data is more expensive than real data because the illusion of short-term cost savings eventually results in higher long-term costs when reality and artifice collide.
Why is real data expensive? Meaningful data is not just a bunch of numbers or tokens; it is the output of some natural process. Producing real data requires costly endeavors like carefully designed experiments or human authors crafting language that is the authentic output of the human authorship process.
When used correctly, as has been done for three decades, synthetic data can provide rigorous benefits over the real data it is based on. For example, real data can be transformed into synthetic data that faithfully preserves all but certain designated “private” properties of the original. This type of synthetic data is more expensive than the original real data, as the additional transformations must be correctly executed by competent professionals in that field.
Recently, however, “synthetic data” has been redefined as merely the output of somebody’s generative model, produced in arbitrarily large quantities at zero incremental cost. This is not real “data” from a natural process; it merely amplifies to an arbitrary degree whatever modeling assumptions, missing elements of reality, choices of training data, biases, etc., went into constructing that generative model. The value of automated generation of such “synthetic data” is independent of the amount produced, as it is all based on exactly the same information and/or misinformation.
Despite the zero incremental cost of generating it, such “synthetic data” will incur large long-term costs. By the time you have built an entire system that is supposed to operate in the real world on this foundation, you will discover that the difference between reality and somebody’s generative model can have serious consequences. For example, some people have advocated the use of such “synthetic data” for the design of self-driving car systems, foreseeably leading to high long-term costs in both dollars and lives.
