Comments for Hubbard Decision Research Blog http://blog.hubbardresearch.com Welcome to the consolidated blog and reader downloads for Doug Hubbard' books Thu, 19 Aug 2010 14:18:23 -0600 http://wordpress.org/?v=2.8.4 hourly 1 Comment on The Measurement Challenge by gchesterton http://blog.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-198 gchesterton Thu, 19 Aug 2010 14:18:23 +0000 http://blog.hubbardresearch.com/?p=491#comment-198 Doug: Here's one for you, since you mention capture-recapture in HTMA. What I’d like to do is estimate the true number of events of type t in a system. I have two databases, A and B, neither of which has complete reporting of these events. I review records from database A and count reports of type t ( I suspect they’re under-reported). Lets say there are 193. I then review database B and count reports of type t. (This database also subject to under-reporting). There are 69. There are fields in the records that allow the analyst to identify events of type t from database A that are the same events as those found in database B. There are 20 that are common to the two counts. Can I use capture-recapture? Any caveats to its use? I come up with an estimate of 665, with a 95% CI of [476, 929]. Doug:
Here’s one for you, since you mention capture-recapture in HTMA. What I’d like to do is estimate the true number of events of type t in a system. I have two databases, A and B, neither of which has complete reporting of these events. I review records from database A and count reports of type t ( I suspect they’re under-reported). Lets say there are 193. I then review database B and count reports of type t. (This database also subject to under-reporting). There are 69. There are fields in the records that allow the analyst to identify events of type t from database A that are the same events as those found in database B. There are 20 that are common to the two counts. Can I use capture-recapture? Any caveats to its use? I come up with an estimate of 665, with a 95% CI of [476, 929].

]]> Comment on What to say when they ask, “Why not 100% CI?” by gchesterton http://blog.hubbardresearch.com/2009/09/what-to-say-when-they-ask-why-not-100-ci/comment-page-1/#comment-197 gchesterton Thu, 05 Aug 2010 17:34:58 +0000 http://blog.hubbardresearch.com/?p=297#comment-197 This is an old thread, but worth reviving for the common misconceptions and for the value of your responses. It's a common problem...dealing with managers who may want "best" point estimates rather than interval estimates...even when they truly do understand the concepts. In these cases where you want to preserve some sense of uncertainty while still providing a single value, then the analyst can resort to the one-sided LCB (or UCB depending on the good/bad context). You may have implied this in your response to the original Q#2. Then at least the analyst can say "I am 90% confident the value is above X million." Usually there is a good/bad direction on the scale, so a UCB or LCB may be sufficient. That said, ignoring uncertainty in either direction can lead to bad (or non-) decisions. This is an old thread, but worth reviving for the common misconceptions and for the value of your responses.
It’s a common problem…dealing with managers who may want “best” point estimates rather than interval estimates…even when they truly do understand the concepts. In these cases where you want to preserve some sense of uncertainty while still providing a single value, then the analyst can resort to the one-sided LCB (or UCB depending on the good/bad context). You may have implied this in your response to the original Q#2. Then at least the analyst can say “I am 90% confident the value is above X million.”
Usually there is a good/bad direction on the scale, so a UCB or LCB may be sufficient. That said, ignoring uncertainty in either direction can lead to bad (or non-) decisions.

]]> Comment on How To Measure Anything Downloads for 1st and 2nd Editions(Registration Required) by dwhubbard http://blog.hubbardresearch.com/questions-about-downloads/comment-page-1/#comment-196 dwhubbard Wed, 04 Aug 2010 02:53:20 +0000 http://blog.hubbardresearch.com/?page_id=44#comment-196 BMH, I gather you haven't read my book yet. In fact, you should find that quite a few of the examples in my book are geared toward government agencies including the EPA, VA, and HUD. And, as with any not-for-profit, thier output must at least be observable in some way directliy or even indirectly. Just like any organization, you have to ask why you care (i.e. what decisions could be different with this knowledge), how you observe the thing you care about, how much you know about it now and so on. Once you answer the questions I pose the book, the rest is a trivial bit of math. Tell me how it works out for you and feel free to post the NFP challenges you encounter in measurement. I think you will find a couple of previous posts in here on exactly that topic. Thanks for your input, Doug BMH,

I gather you haven’t read my book yet. In fact, you should find that quite a few of the examples in my book are geared toward government agencies including the EPA, VA, and HUD. And, as with any not-for-profit, thier output must at least be observable in some way directliy or even indirectly. Just like any organization, you have to ask why you care (i.e. what decisions could be different with this knowledge), how you observe the thing you care about, how much you know about it now and so on. Once you answer the questions I pose the book, the rest is a trivial bit of math.

Tell me how it works out for you and feel free to post the NFP challenges you encounter in measurement. I think you will find a couple of previous posts in here on exactly that topic.

Thanks for your input,
Doug

]]> Comment on Facilitating Calibrated Estimates by gchesterton http://blog.hubbardresearch.com/2008/12/facilitating-calibrated-estimates/comment-page-1/#comment-195 gchesterton Mon, 02 Aug 2010 20:11:36 +0000 http://blog.hubbardresearch.com/?p=237#comment-195 Doug: In an answer to KDR, I would suggest some sort of optimization, whereby the individuals whose estimates were best (a posteriori) are given higher weightings. Doug:
In an answer to KDR, I would suggest some sort of optimization, whereby the individuals whose estimates were best (a posteriori) are given higher weightings.

]]> Comment on How To Measure Anything Downloads for 1st and 2nd Editions(Registration Required) by hackettbmh http://blog.hubbardresearch.com/questions-about-downloads/comment-page-1/#comment-194 hackettbmh Mon, 02 Aug 2010 14:35:22 +0000 http://blog.hubbardresearch.com/?page_id=44#comment-194 I work at a non-profit in Measurement. Non-profits have decidedly softer measures. I was wondering if you had any pointers on non-profit measurement, or if you could direct me to any good resources on the subject? Best, BMH I work at a non-profit in Measurement. Non-profits have decidedly softer measures. I was wondering if you had any pointers on non-profit measurement, or if you could direct me to any good resources on the subject?

Best,
BMH

]]> Comment on The Measurement Challenge by rolandk http://blog.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-193 rolandk Sun, 01 Aug 2010 15:29:52 +0000 http://blog.hubbardresearch.com/?p=491#comment-193 In fact I already did a simple experiment, translating the Chicago Piano Tuner problem into the Viennese Hair Salon Monte Carlo Simulation http://objektorient.blogspot.com/2010/08/freelibre-and-open-source-aie-models.html I hope its okay if I copyleft this even if the inspiration came from you, as the bin-slicing excel formula does. In fact I already did a simple experiment, translating the Chicago Piano Tuner problem into the Viennese Hair Salon Monte Carlo Simulation

http://objektorient.blogspot.com/2010/08/freelibre-and-open-source-aie-models.html

I hope its okay if I copyleft this even if the inspiration came from you, as the bin-slicing excel formula does.

]]> Comment on The Measurement Challenge by dwhubbard http://blog.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-192 dwhubbard Sun, 01 Aug 2010 12:46:37 +0000 http://blog.hubbardresearch.com/?p=491#comment-192 Roland, So then we must compare the harm of false reliance on induction to the harm of false rejection of induction. I can never say this enough: What was the rate and magnitude of error before using a particular method and the rate and magnitude of errors after using it? All of the empirical evidence says that unaided human intuition is easily outperformed by even simple modeling. Remember, all models are wrong. They all have error. The question is whether your previous model (intution) really had less error. The eveidence says no. Yes, feel free to tell me more about your measurement study. Doug Hubbard Roland,

So then we must compare the harm of false reliance on induction to the harm of false rejection of induction. I can never say this enough: What was the rate and magnitude of error before using a particular method and the rate and magnitude of errors after using it? All of the empirical evidence says that unaided human intuition is easily outperformed by even simple modeling.

Remember, all models are wrong. They all have error. The question is whether your previous model (intution) really had less error. The eveidence says no.

Yes, feel free to tell me more about your measurement study.

Doug Hubbard

]]> Comment on The Measurement Challenge by rolandk http://blog.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-191 rolandk Sun, 01 Aug 2010 12:30:42 +0000 http://blog.hubbardresearch.com/?p=491#comment-191 Doug, today I tried and drafted now several arguments why immeasurables exist, but no one seems to be right. The only thing I could say is that a false reliance on induction could harm you. Because the future might change. But this is in the TFoRM book. (btw. I think it would be an interesting experiment if you would try to convince us that immesurables exist.) More than keeping up with the challenge I would like to send you eventually some models our little measurement study group here in Italy will come up with. Next week is our second meeting, I will calibrate the guys and then we identify some problems to work on. Many thanks, Roland Doug, today I tried and drafted now several arguments why immeasurables exist, but no one seems to be right.
The only thing I could say is that a false reliance on induction could harm you. Because the future might change. But this is in the TFoRM book.
(btw. I think it would be an interesting experiment if you would try to convince us that immesurables exist.)

More than keeping up with the challenge I would like to send you eventually some models our little measurement study group here in Italy will come up with. Next week is our second meeting, I will calibrate the guys and then we identify some problems to work on.
Many thanks, Roland

]]> Comment on The Measurement Challenge by dwhubbard http://blog.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-190 dwhubbard Mon, 26 Jul 2010 18:13:26 +0000 http://blog.hubbardresearch.com/?p=491#comment-190 Roldand, Why would it necessarilly have no credibility? Remember, the real test is whether it outperforms your intuion. If you are both equally well calibrated but they turn out to be wrong 30% of the time in a large number of trials, but you turn out to be wrong 40% of the time, then its an improvement. Heed Voltaire when he says the perfect is the enemy of the good. Measurement is about uncertainty reduction and if a model - with all its flaws - is right more often than your previous model (intuition) then it was a measurement. Is your question about whether such a model could concevaibly outperform intution of the average sports expert? Why do you think it couldn't? It's all about results and if the results are extremely unlikely to be do to chance alone then the results are informative. Doug Roldand,

Why would it necessarilly have no credibility? Remember, the real test is whether it outperforms your intuion. If you are both equally well calibrated but they turn out to be wrong 30% of the time in a large number of trials, but you turn out to be wrong 40% of the time, then its an improvement. Heed Voltaire when he says the perfect is the enemy of the good. Measurement is about uncertainty reduction and if a model – with all its flaws – is right more often than your previous model (intuition) then it was a measurement.

Is your question about whether such a model could concevaibly outperform intution of the average sports expert? Why do you think it couldn’t? It’s all about results and if the results are extremely unlikely to be do to chance alone then the results are informative.

Doug

]]> Comment on The Measurement Challenge by rolandk http://blog.hubbardresearch.com/2010/03/the-measurement-challenge/comment-page-1/#comment-189 rolandk Sun, 25 Jul 2010 23:20:35 +0000 http://blog.hubbardresearch.com/?p=491#comment-189 Just for curiosity -as i want to know if such prediction is credible at all: UBS disclosed likelihoods of a teams chance to reach the next round. You can see them on page 16 of UBS Investors Guide. http://www.ubs.com/2/e/medlib/wmr/IGWM_spez_2010_en.pdf In fact the article at page 14 explains some of the model, citation: “As in our previous studies, we rely exclusively on three factors to estimate the different winning probabilities: 1) past performance; 2) whether or not a team is a host nation; and 3) an objective quantitative measure that assesses the strength of each team three months before the start of the World Cup. Socioeconomic factors like population size or GDP growth have been proven to have no explanatory power when it comes to forecasting the performance of a specific team.” Just for curiosity -as i want to know if such prediction is credible at all:

UBS disclosed likelihoods of a teams chance to reach the next round. You can see them on page 16 of UBS Investors Guide. http://www.ubs.com/2/e/medlib/wmr/IGWM_spez_2010_en.pdf
In fact the article at page 14 explains some of the model, citation: “As in our previous studies, we rely exclusively on three factors to estimate the different winning probabilities:
1) past performance;
2) whether or not a team is a host nation;
and
3) an objective quantitative measure that assesses the strength of each team three months before the start of the World Cup. Socioeconomic factors like population size or GDP growth have been proven to have no explanatory power when it comes to forecasting the performance of a specific team.”

]]>