C O M M E N TA R Y

Serum proteomics profiling—a young

technology begins to mature

During the three years since the US National reproducibility remain unresolved. This article
Cancer Institute–Food and Drug Administration outlines some possible improvements in experi-
(NCI-FDA) proteomics group published their mental design and data analysis that might help
seminal1 (but flawed2–4) study using mass in this regard.
spectrometry to profile the serum proteome
of ovarian cancer patients, more than 60 pub- Potential problems
lished studies have applied similar technology It is important to recognize the provisional
to a wide range of cancers and other diseases. nature of the conclusions of most serum pro-
Although some studies have looked directly at teomic studies carried out to date. In most
tumor tissue, most have targeted easily accessible cases where the protein peaks have been iden-
fluids such as serum. A few studies used high- tified, they have turned out to be well-known,
resolution instruments with a matrix-assisted acute-phase proteins. Critics of this application
P.M. Motta & S. Makabe/Photo Researchers, Inc.
laser desorption and ionization (MALDI) ion of mass spectrometry have claimed that it is
source and a time-of-flight (TOF) ion detector. inherently limited in its depth of coverage, with The seminal NCI-FDA study used mass
The majority, however, have relied on lower res- a dynamic range that prevents detection of low- spectrometry to analyze the serum proteome of
olution instruments and surface-enhanced laser abundance proteins12,13. As for predictive pat- patients with ovarian cancer, cells of which are
shown above in a scanning electron micrograph
desorption and ionization (SELDI), a MALDI terns, no two independent studies have found
(magnification: 414×).
variant that uses commercially prepared chro- the same patterns to date. Moreover, none of the
matographic surfaces (ProteinChips; Ciphergen, patterns has been validated in an independent
Fremont, CA, USA) to separate proteins. study by another laboratory. In one study, both contrasts of interest. Similar problems appear to
These studies have a common goal: to identify the sensitivity and the specificity of the pattern have affected the NCI-FDA studies4,16.
biomarker patterns in the proteome that can be declined significantly when samples were pro- In another MALDI study of the serum pro-
used for diagnosis, prognosis or monitoring of cessed in the same laboratory after a delay of teome of 50 cancer patients carried out by our
disease. The preponderance of evidence from several months14. groups at M.D. Anderson15, hierarchical clus-
these studies seems to suggest that proteomic Our experience at the M.D. Anderson tering produced two clusters. In this case, the
profiling is able to detect anonymous protein Cancer Center suggests that it can be difficult sample collection protocol had been changed
peaks that are expressed differently in cancer to obtain stable, reproducible mass spectrom- after the first 20 samples had been collected;
patients than in healthy individuals5–7. The etry results over time and across laboratories. the clusters matched the sample collection
studies also provide evidence that we may be In one unsuccessful study, SELDI experiments protocols. The change in protocols resulted in
able to discover patterns that are useful for pre- were performed to analyze the serum proteome reproducible changes in the serum proteome,
diction of the presence or absence of disease8–11. from 247 patients with five different subtypes and these changes were significantly larger than
For serum proteomics to realize its full poten- of cancer, with 40, 60, 65, 62 and 20 patients those resulting from different histological sub-
tial, however, several problems in sensitivity and with each subtype. Simple hierarchical cluster- types of cancer15.
ing (average clustering with distance based on The source of these problems appears to be
Kevin R. Coombes, Jeffrey S. Morris, Jianhua Pearson correlation) of the data produced six inextricably linked to the promise of the tech-
Hu and Keith A. Baggerly are at the Section of clusters. Unfortunately, the clusters matched nology. Mass spectrometry can be exquisitely
Bioinformatics, Department of Biostatistics and the run dates of the samples, not the biological sensitive to changes in the part of the proteome
Applied Mathematics, The University of Texas subtypes of cancer15. The same clustering was that it can measure accurately. Differences in
M.D. Anderson Cancer Center, 1515 Holcombe observed using repeated runs of a quality con- sample collection or sample handling affect the
Blvd., Box 447, Houston, Texas 77030, USA trol sample, which was run concurrently with proteome, to a degree that can dominate bio-
and Sarah R. Edmonson is in the Department the experimental samples. Because some of the logical changes. In addition, as with many sen-
of Family and Community Medicine, Baylor biological subtypes were confounded with the sitive instruments, mass spectrometers can be
College of Medicine, Houston, Texas 77098, USA. run order, it was impossible in this data set to temperamental. They provide the most accurate
e-mail: kcoombes@mail.mdanderson.org separate technological effects from biological and highest resolution results over a very small

NATURE BIOTECHNOLOGY VOLUME 23 NUMBER 3 MARCH 2005 291

subset of their mass range, where they have extreme, the NCI-FDA proteomics group per- used to uncover proteomic expression patterns
been carefully focused. Relatively small changes forms minimal preprocessing, and they allow the linked to a disease state. Careful studies are now
to the operating conditions can be amplified to intensity at every measured mass-to-charge (m/ underway to develop reproducible protocols. An
produce fairly large differences in mass spectra, z) value to be a potential feature for distinguish- increased awareness of the importance of sound
making it difficult to maintain consistent, repro- ing cancer patients from healthy controls1,23,24. experimental design is enhancing the ability
ducible results. Because their experimental design confounded of studies to yield reproducible results. Rapid
biology with technology2–4,16, their approach is improvements are being made in developing
Potential solutions not yet firmly established. At the other extreme, methods to analyze the data produced by mass
Reproducibility can be enhanced in several a majority of published studies perform pre- spectrometry protein profiling experiments. The
© 2005 Nature Publishing Group http://www.nature.com/naturebiotechnology

ways: by improving instrumentation, experi- processing and peak detection using software next step will be to show that these improved
mental protocols, study design or analysis tools. from Ciphergen, the manufacturer of the SELDI protocols, designs and analytical tools produce
Although improved instruments might help, the instrument25. In our opinion, the Ciphergen results that are sufficiently reliable and repro-
studies conducted to date using higher resolu- software is overly conservative when peak find- ducible to change the way patients are treated.
tion MALDI instruments have not convincingly ing, and its baseline correction algorithm pro-
1. Petricoin, E.F. et al. Lancet 359, 572–577 (2002).
demonstrated that they produce more robust duces biased estimates of peak heights. These 2. Sorace, J.M. & Zhan, M. BMC Bioinformatics 4, 24
proteomics patterns than the SELDI approach. algorithmic weaknesses can reduce the effective (2004).
3. Baggerly, K.A., et al. Bioinformatics 20, 777–785
This may change as the technology evolves. sensitivity of the instrument below its true capa-
(2004).
Performing peptide profiling instead of protein bilities and can hamper its reproducibility. 4. Baggerly, K.A., et al. Signal in noise: evaluating reported
profiling is promising because it concentrates Researchers have been actively developing reproducibility of serum proteomics tests for ovarian
cancer. J. Natl. Cancer Inst., 97, in press (2005).
all measurements in a small region of extremely better methods for processing mass spectra. 5. Vlahou, A. et al. Clin. Chem. 50, 1438–1441 (2004).
high mass resolution17,18. In addition, high-end Yasui et al.26,27 have used a super smoother to 6. Pusztai, L. et al. Cancer 100, 1814–1822 (2004).
Fourier transform mass spectrometry is being find peaks, but had to settle for binary indica- 7. Lin, Z., Jenson, S.D., Lim, M.S. & Elenitoba-Johnson,
K.S. Mod. Pathol. 17, 670–678 (2004).
evaluated for its reproducibility in serum pro- tors of presence or absence in lieu of quanti- 8. Zhang, Z. et al. Cancer Res. 64, 5882–5890 (2004).
teomic profiling19. fication. Our group28 introduced an iterative 9. Wadsworth, J.T. et al. Clin. Cancer Res. 10, 1625–1632
(2004).
The Early Detection Research Network method to perform simultaneous background
10. Soltys, S.G. et al. Clin. Cancer Res. 10, 4806–4812
(EDRN) is currently conducting a multi- correction and peak finding, but this method (2004).
institutional study to develop and validate required manual adjustment of numerous 11. Xiao, Z. et al. Cancer Res. 64, 2904–2909 (2004).
12. Diamandis, E.P. Expert Rev. Mol. Diagn. 4, 575–577
protocols to produce reproducible mass parameters. Qu et al.29 have applied wavelets for (2004).
spectra20. Their initial report computed the data reduction. Because the wavelet coefficients 13. Diamandis, E.P. Mol. Cell Proteomics 3, 367–378
coefficient of variation in mass location, sig- do not directly correspond to physical quanti- (2004).
14. Rogers, M.A. et al. Cancer Res. 63, 6971–6983
nal-to-noise ratio and normalized intensity for ties, however, they are difficult to interpret. (2003).
three protein peaks, concluding that between- We also used wavelets for noise reduction30, 15. Hu, J., et al. Brief. Funct. Genomic. Proteomic. 3, 322–
331 (2005).
laboratory reproducibility was as good as especially in concert with the use of the mean
16. Baggerly, K.A., et al. Endocr. Relat. Cancer 11, 583–584
within-laboratory reproducibility21. It is still spectrum to borrow strength across spectra31. (2004).
unclear, however, if reproducibility of the loca- Our studies benefited from a computer model 17. Villanueva, J. et al. Anal. Chem. 76, 1560–1570
(2004).
tion and intensity of a few peaks is enough to based on the physics of a mass spectrometer, 18. Koomen, J.M. et al. Rapid Commun. Mass Spectrom.
ensure the reproducibility of the entire spec- which used a simple model of the detector32. 18, 2537–1848 (2004).
trum. It is possible that more global measures In another approach, Sauve and Speed33 have 19. Johnson, K.L., et al. Anal. Chem. 76, 5097–5103
(2004).
of reproducibility (that take into account all used dynamic programming to improve calibra- 20. Grizzle, W.E. et al. Dis. Markers 19, 185–195 (2004).
or most peaks) will be required. tion across multiple spectra and morphological 21. Semmes, O.J. et al. Clin. Chem. 51, 102–112 (2005).
22. Ransohoff, D.F. Nat. Rev. Cancer 4, 309–314 (2004).
The need for better study design and experi- filters for baseline correction.
23. Petricoin, E.F. 3rd et al. J. Natl. Cancer Inst. 94, 1576–
mental design for proteomic profiling has been Elsewhere, Malyarenko et al.34 have 1578 (2002).
discussed elsewhere15,22. Epidemiologists and approached mass spectra from the vantage of 24. Conrads, T.P. et al. Endocr. Relat. Cancer 11, 163–178
(2004).
statisticians have known the necessary design time series. They obtained better estimates of 25. Fung, E.T. & Enderwick, C. Biotechniques Suppl, 34–41
principles for a long time. It often seems, how- baseline by realizing that its primary source (2002).
ever, that these principles are forgotten in the resided in the physics of the ion detector. They 26. Yasui, Y. et al. J. Biomed. Biotechnol. 2003, 242–248
(2003).
excitement that accompanies the first applica- also used deconvolution filters to smooth the 27. Yasui, Y. et al. Biostatistics 4, 449–463 (2003).
tions of a new technology. The spectrum of sub- signal and obtain better resolution of mass 28. Coombes, K.R. et al. Clin. Chem. 49, 1615–1623
(2003).
jects in the study, both with and without disease, peaks. We expect that incorporating knowledge
29. Qu, Y. et al. Biometrics 59, 143–151 (2003).
must reflect the actual population. Validation of the physical and chemical properties of mass 30. Coombes, K.R. et al. Improved peak detection and
data sets must be independent and large. Care spectrometry instruments will lead to further quantification of mass spectrometry data acquired from
surface-enhanced laser desorption and ionization by
must be taken to avoid confounding biological improvements in analytical methods. denoising spectra with the undecimated discrete wavelet
variables of interest (e.g., presence or absence transform. Proteomics, in press (2005).
of disease) with technological factors (e.g., Conclusions 31. Morris, J.S., Coombes, K.R., Koomen, J., Baggerly, K.A.
& Kobayashi, R. Bioinformatics, published online 26
sample collection or run order) that might bias Mass spectrometry profiling of the serum pro- January 2005.
the results. Appropriate measures to minimize teome remains an exciting tool with the poten- 32. Coombes, K.R., et al. Understanding the characteristics
of mass spectrometry data through the use of simulation.
confounding and bias include running blinded tial to transform medicine. As often happens
Cancer Informatics, in press.
tests, randomization and blocking. with a young technology, the spectacular claims 33. Sauve, A.C. & Speed, T.P. Normalization, baseline cor-
The current state of the art in the analysis of of early studies have not held up under closer rection and alignment of high-throughput mass spec-
trometry data Proceedings Gensips 2004, in press
serum proteome profiling experiments allows scrutiny. Nevertheless, numerous studies have (2005).
considerable room for improvement. At one provided evidence that this technology can be 34. Malyarenko, D.I. et al. Clin. Chem. 51, 65–74 (2005).

292 VOLUME 23 NUMBER 3 MARCH 2005 NATURE BIOTECHNOLOGY