Evaluating the Repeatability and Differences between ICP Tests

[Thales]

Evaluating the Repeatability and Differences between ICP Tests | Reef Builders | The Reef and Saltwater Aquarium Blog

By Sanjay Joshi and Richard Ross. ICP testing has become more mainstream in the past few years since it was introduced in the hobby almost a decade ago, but questions about the utility of the testing…

reefbuilders.com

Tear it up.

 

[Dan_P]

Evaluating the Repeatability and Differences between ICP Tests | Reef Builders | The Reef and Saltwater Aquarium Blog

By Sanjay Joshi and Richard Ross. ICP testing has become more mainstream in the past few years since it was introduced in the hobby almost a decade ago, but questions about the utility of the testing…

reefbuilders.com

Tear it up.

Agrees with the investigations we (@taricha, @Rick Mathew ) performed. For elements with concentrations above 1 ppm, variation is small and vendor consensus is not bad. Below 1 ppm, variation increases with decreasing concentration and vendor consensus can be very poor. ICP-MS is more sensitive (detects lower concentration elements) than ICP-OES but is not necessarily more accurate. For many vendors, there seems to be elements that are poorly measured. This points to ICP vendors using a non-validated analytical method. If you have not notice, ICP vendors do not provide the accuracy of the results they sell you.

All this sounds really bad until you realize that the acceptable concentration range for each element is up in the air. Maybe large variation in ICP results are acceptable in this situation. ICP results might not be good enough for trending or judging how much to dose trace elements, but might be useful for detecting high levels of toxic elements, or detecting the presence or absence of trace elements.

 

[ScottD]

So the big take away for me is they are pretty consistent on their results but not precise in comparison to each other. So the old adage remains true: don’t chase numbers, chase stability.

 

[Randy Holmes-Farley]

One thing discussed briefly in the article but not really answered is how accurate does one really need trace metal levels to be.

Suppose that iron is adequately bioavailable without being toxic in the extremely wide range from 0.001 ppm to 0.1 ppm (it may actually be much wider).

In that case, if one targets 0.01 ppm, then the measured value can be off by a factor of 10x up or down, and still be adequate to ensure the level is high enough but not too high.

 

[taricha]

I love this so much.
especially for this part...
"15 samples were collected for testing within the span of time it took to fill all the sample tubes with water. The samples were shipped arbitrarily, five at a time, on Monday, Wednesday, and Friday of the same week."

We can think of three sorts of variations in our hobby ICP testing (for major elements that are stable).
1. The variation you get from a vendor running replicate samples over and over (back-to-back variability)
2. The variation you get when a vendor runs a replicate sample on a different day, after the machine has gone through a cleaning/recalibration routine. (day to day variability)
3. The variation from choosing vendor A vs B vs C. (vendor to vendor variability)

Lots of people have tested 1 and 3, but the data on 2, day to day variability is super thin. We know 1 - back to back variability - is small. That's been apparent since the Ross & Maupin Triton article years ago, and confirmed many times since.
We know 3 is significantly larger. Sanjay's recent reefbuilders article
gives a great tour of how different results of one sample can look when you send it to different vendors. Many others have noticed (and griped) about the same thing.

What @Rick Mathew , and @Dan_P and I have wondered about is number 2 - day to day variability. What if the larger vendor to vendor variation was really just the day to day variation? I mean if you send replicate samples to the same vendor on different days, does it just look like you sent it to different vendors?
The relevance here is trending elements. We say just stick with one vendor for trending, well what if that vendor looks like a different vendor from one day to the next?

This is a really good data set for answering the question because it includes the back-to-back and day-to-day variability in the vendor data.
Let's look through the first tiers of elements.
This chart is vendors A, B, and C with the means and max/min for each element. All normalized to the combined average across 3 vendors set to 1.

First, you can see that the question of what variations matter the most is clearly answered. Even the day-to-day recalibration/cleaning routines of the vendors don't add enough variation to account for differences between vendors. So systematic variations between vendors is the largest source of variation in data that we see. (The article demonstrates this rigorously by the statistical Tukey's test.)

Thus, the advice to stick to one vendor to trend elements is well-supported.

Secondly, what would people use these particular results in this chart for?
two things: salinity and check major ion balance (Chloride vs Sulfate etc).
The salinity measures look good. The hobbyist can probably live with Chloride +-4% and Salinity +-3% as limits on how well salinity can be measured.
So if your salinity actually was 35ppt, accept that you can get 34 to 36 from ICP, and don't adjust based on expecting tighter than that.
For major ion balance, the Chloride +-4% and Sulfate of +-10% are probably okay too. I don't think anybody would decide to try to adjust ionic balance based on sulfate vs chloride being off by ~10% from an ideal balance.

Later I'll post a look at the titration elements: Ca, Mg, and K.

 

[taricha]

relevant to previous post @Thales do the dates/times on the reports allow you to confirm that each vendor actually ran samples on different days, or did any vendor hold them all and run them on the same day in possibly one session?

 

[jda]

Other than the regular stuff that people might complain about like "their" company uses MS or something and they do not know that it does not matter for this, there is nothing to tear up. The maths is the maths.

Do you plan on doing the same thing with the compounds that people care so much about? ...the results that you get back that were not done with the plasma and require other tools and testing to be used? These seem to be where the most variation comes and it is not plasma related at all, but it is part of the service that they offer.

 

[Rick Mathew]

One thing discussed briefly in the article but not really answered is how accurate does one really need trace metal levels to be.

Suppose that iron is adequately bioavailable without being toxic in the extremely wide range from 0.001 ppm to 0.1 ppm (it may actually be much wider).

In that case, if one targets 0.01 ppm, then the measured value can be off by a factor of 10x up or down, and still be adequate to ensure the level is high enough but not too high.

I totally agree!...


From my perspective knowing the acceptable range needed from a measurement is the second half of the equation. Knowing the accuracy and precision of a gage is good, but that is only the first half of the equation and is not as useful if you do not have a defined acceptable range. This defined range along with the measured accuracy and precision of the test can provide the level of confidence you can place on taking action/ or not on the measured results.

As in your example the range is quite wide and therefore allows for a greater amount of variability in the measurement. On the other hand if the allowable range were .001ppm to .010ppm the measurement variability would need to be much lower for one to have confidence in taking any action.


In the article published by @Dan_P ,@taricha and myself “How We Use ICP-OES Results Of Unknown Accuracy And Precision” under the section “FINDING THE GAGE VARIATION OF A MEASUREMENT SYSTEM”, we discuss this in some detail


https://www.reef2reef.com/ams/how-we-use-icp-oes-results-of-unknown-accuracy-and-precision.862/

 

[Thales]

relevant to previous post @Thales do the dates/times on the reports allow you to confirm that each vendor actually ran samples on different days, or did any vendor hold them all and run them on the same day in possibly one session?

I am flat on my back sick so I hope this makes sense!

I know that they were not all done on the same day, but I would have to check to confirm if they were all on different days. And, I suppose the reality is that we really only have the date they gave the results and no real way to know when the samples were run.

 

[Thales]

Other than the regular stuff that people might complain about like "their" company uses MS or something and they do not know that it does not matter for this, there is nothing to tear up. The maths is the maths.

Thanks!

Do you plan on doing the same thing with the compounds that people care so much about? ...the results that you get back that were not done with the plasma and require other tools and testing to be used? These seem to be where the most variation comes and it is not plasma related at all, but it is part of the service that they offer.

Dunno! It is becomming harder to tell how the vendors are testing for what, and from my sick couch that seems troubling.

I am also feeling like all of the testing and number crunching we have all been doing keep confirming the same thing the original write ups concluded (with some interesting details for sure), so I am not sure how much more we should be paying these companies to test their products unless there is some shift in methodology.

But I am sick and may not be thinking clearly.

 

[jda]

My goal is for everybody to understand what goes through the plasma and what gets done in other ways. Then, understand that the other ways have wider/tighter ranges of acceptability. The thread from some of the Gs above titled something like Hobby Grade Test Kits are Just as Good hit on this some... some of these companies are just using Hannah Eggs to test for some of these things, but Christoph is the only one who has specifically stated.

Do you have the compound numbers from these tests? I could run them if you want to send them over. It would be nice to see variance though CI, CV, ANOVA or a SuperNova with these numbers too. It could be broken down in elements when Plasma was used and then the rest...

...not that either of you need more work, but it could be a Part II to your article.

 

[Rick Mathew]

Thanks!

Dunno! It is becomming harder to tell how the vendors are testing for what, and from my sick couch that seems troubling.

I am also feeling like all of the testing and number crunching we have all been doing keep confirming the same thing the original write ups concluded (with some interesting details for sure), so I am not sure how much more we should be paying these companies to test their products unless there is some shift in methodology.

But I am sick and may not be thinking clearly.

I think you are thinking quite clearly!

 

[Miami Reef]

I don’t use ICP often. If I have mysterious coral issues, I’ll send a sample. As long as nothing is dangerously elevated, I’m good.

I don’t watch for specific values. I just make sure nothing is extremely out of range that can explain the coral issue.

I use balling part C in addition to my 2 part, so my ions remain balanced when maintaining proper salinity. I change water regularly.

I never had ICP tell me something enlightening before.

 

[Dan_P]

One thing discussed briefly in the article but not really answered is how accurate does one really need trace metal levels to be.

Suppose that iron is adequately bioavailable without being toxic in the extremely wide range from 0.001 ppm to 0.1 ppm (it may actually be much wider).

In that case, if one targets 0.01 ppm, then the measured value can be off by a factor of 10x up or down, and still be adequate to ensure the level is high enough but not too high.

Exactly. This concept is something you and @Rick Mathew have been emphasizing for some time.

If you don’t know what the acceptable range is for the analyte of interest, you cannot determine whether the test result variation meets your testing needs. For some elements in this hobby, a random number generated around an average ocean water value will provide a sufficiently useful analysis.

 

[areefer01]

If you don’t know what the acceptable range is for the analyte of interest, you cannot determine whether the test result variation meets your testing needs.

What happens when ICP is used to calculate the dosing amount for additive based systems (those that use them)? Not trying to stir the pot but it is something that always crosses my mind when I read these things and lads who implement them.

I think anyone who uses ICP should probably stick with the same vendor anyway. Assuming they have good protocols in place it gives data retention more value for trend analysis. At least this is what I do.

Only question I had after reading the assessment is if the ICP vendors chosen use the same type (if that is the correct word) of machine or format. And if it matters. OES, MS, machine manufacturing, etc. When looking at data things can get a bit precarious.

 

[Dan_P]

What happens when ICP is used to calculate the dosing amount for additive based systems (those that use them)? Not trying to stir the pot but it is something that always crosses my mind when I read these things and lads who implement them.

This is precisely the issue, using data with unknown accuracy and precision to judge how much stuff to add. There might be a safety valve though. Since there does not seem to be an issue of trace element accumulation occurring in the hobby, could it be that the average hobby aquarium is quickly eliminating trace elements from the water by biotic and abiotic processes?

Only question I had after reading the assessment is if the ICP vendors chosen use the same type (if that is the correct word) of machine or format. And if it matters. OES, MS, machine manufacturing, etc. When looking at data things can get a bit precarious.

Excellent questions, but how would we use that knowledge if we had it? The only thing we know for certain is that ICP vendor methods, regardless of the detection method, are not sufficiently reliable for them to predict how accurate their test results are.

 

[areefer01]

This is precisely the issue, using data with unknown accuracy and precision to judge how much stuff to add. There might be a safety valve though. Since there does not seem to be an issue of trace element accumulation occurring in the hobby, could it be that the average hobby aquarium is quickly eliminating trace elements from the water by biotic and abiotic processes?

I agree. And to complicate matters what values are used as the standard and why are they chosen.

Excellent questions, but how would we use that knowledge if we had it? The only thing we know for certain is that ICP vendor methods, regardless of the detection method, are not sufficiently reliable for them to predict how accurate their test results are.

On a hobbyist level maybe nothing. If I was trying to land on the moon maybe everything. I guess my question was a bit vague and not my intent. To me it is a variable because one may be able to detect it and the other may not. Or at different levels. The initial tables with the results from vendor A, B, and C we can see that A, and C do not detect Cobalt (for topic of discussion) yet vendor B does. It could be that A and B are MS and C OES. I don't know but there is a difference. It very well could be nothing and just me grasping at the differences in the data.

Thank you for the reply btw.

 

[Nburg's Reef]

Thanks Rich (and Sanjay and the beefer purse) for this. Sort of makes me feel good about my stance that I’ll not pay for an ICP or give it a lot or credibility until 1) there is a standardized/vetted method and 2) QC data. When I get ICP air samples, I know they followed NIOSH 7300 because they said they did, show their every 5th QC sample results and they have credentials that they get third party audits. They also cost about $150 per test, which makes sense given the amount of effort that goes into accuracy and precision.

 

[taricha]

I know that they were not all done on the same day, but I would have to check to confirm if they were all on different days. And, I suppose the reality is that we really only have the date they gave the results and no real way to know when the samples were run.

Thanks - when you feel likeit. I would like to know if each vendor represented had at least some of the results on different days. And from what I've seen, the dates on the reports usually fit the timelines of when they got the samples and when we get results. I don't think there's any shenanigans there.
Some vendors don't run their machine every day. Some vendors get shipments coming in all lumped together - so tests sent on different days may arrive on the same day.

 

[taricha]

Do you plan on doing the same thing with the compounds that people care so much about? ...the results that you get back that were not done with the plasma and require other tools and testing to be used?

You can see the results in the article for Iodine, Silica and Phosphorus. Three that we have good chemical tests for and ICP is hit or miss on.
What makes you think vendors are doing undeclared chemical tests and passing them off as ICP?