Moving targets and a insufficient database search in a systematic review?
The "systematic" review of Walton R et al. [1] appears to disregard a
considerable body of knowledge, including controlled trials concerning insulin dosage
recommendation systems [2-8], and illustrates the problems a systematic review may run
into if the research question is not well enough defined in advance.
First, the review states the aim to "identify all comparative studies in which
computers were used to help determine the most appropriate drug dose" and in the
original research protocol authors state to include all studies where participants are
"patients receiving drug therapy based on advice from a computer" [9]. Thus,
according to their own protocol, authors would clearly have to include studies where
advice is given directly to patients or to nurses, such as in some software applications
for diabetes management [2-8]. In the abstract of their review however, authors speak of
"computers giving advice to clinicians" [1] (apparently excluding advice
to patients and direct computer-assisted drug administration), but in the article body
they broaden this concept again to include studies in which computers administer drugs
directly to the patient (but inconsequentially not to those where drug advice is
given to the patient). One would at least have expected some discussion for the reason to
exclude studies where advice is given to the patient, especially in the light of the fact
that insulin treatment in diabetes (having clear physiological parameters) should be a
very nice model to demonstrate the power of computerised drug advice and also because
contamination effects (the administering clinician treating both patients from control and
intervention groups learns from the computer advice of the intervention group) cannot
occur in these studies.
Secondly, the authors used an insufficient search strategy, which may have led to
non-retrieval of studies (e.g [10]) and potential bias. The use of "prescr*" as
a search term for the broad concept of drugs (including medication administered by
computer-controlled infusion pumps) is a major flaw, leading to non-retrieval of at least
261200 MEDLINE articles which deal with pharmaceutical preparations but do not use the
word stem "prescr*". Equally insensitive is the usage of the search term
comput*, (for example, at least 9967 MEDLINE articles use the word "software" or
"expert system" without using " comput*"). To rely on the MeSH term
"Computer-Assisted Decision Making" alone (which was introduced as late as 1987)
is a rather insensitive approach for this broad research question, again especially in the
context of computer-controlled pumps.
Thirdly, the authors claimed to include other research designs such as interrupted time
series, but confusingly this is not reflected in the search strategy "randomised
controlled trial OR random allocation OR double blind method", which appears to
target primarily randomised controlled trials.
In summary, the authors seem to have pursued a moving target, originally having in mind
computerised advice to improve "prescribing practice" (as reflected in the title
of the original research protocol [9]) and subsequently broadening this concept to
computer support for determining drug dose including computer-controlled pumps (which is a
much broader and different concept). Their aim of this review was to determine
"whether there is clear evidence that computerised advice on drug dosage is
beneficial"[9], therefore evidence regarding computer-advice given directly to
patients should have been included in this review (even if this group of studies would
perhaps have required subgroup analysis).
These problems should illustrate the importance of defining a clear search strategy in
advance (which has not been done in the protocol of this particular systematic review [9])
and the importance of consulting an experienced information broker before conducting a
literature search.
Walton R, Dovey S, Harvey E, Freemantle N. Computer support for determining drug dose:
systematic review and meta-analysis. BMJ 1999 Apr 10;318(7189):984-990
Chiarelli F, Tumini S, Morgese G, Albisser AM. Controlled study in diabetic children
comparing insulin-dosage adjustment by manual and computer algorithms. Diabetes Care 1990
Oct;13(10):1080-4
Peters A, Kerner W. Analytisches Design und klinische Anwendung eines lernfahigen
Regelsystems fuer die Pharmakotherapie mit Insulin [Analytic design and clinical
application of an intelligent control system for pharmacotherapy with insulin--2]. Biomed
Tech (Berl) 1996 Mar;41(3):42-53
Ryff-de Leche A, Engler H, Nutzi E, Berger M, Berger W. Clinical application of two
computerized diabetes management systems: comparison with the log-book method. Diabetes
Res 1992 Mar;19(3):97-105
Schrezenmeir J, Achterberg H, Bergeler J, Kustner E, Stumer W, Hutten H, Beyer J.
Controlled study on the use of hand-held insulin dosage computers enabling conversion to
and optimizing of meal-related insulin therapy regimens. Life Support Syst 1985;3 Suppl
1:561-7
Strack T, Bergeler J, Beyer J, Hutten H. Computer assisted conventional insulin therapy.
Life Support Syst 1985;3 Suppl 1:568-72
Ambrosiadou BV, Goulis DG, Pappas C. Clinical evaluation of the DIABETES expert system
for decision support by multiple regimen insulin dose adjustment. Comput Methods Programs
Biomed 1996 Jan;49(1):105-15
Peterson CM, Jovanovic L, Chanoch LH. Randomized trial of computer-assisted insulin
delivery in patients with type I diabetes beginning pump therapy. Am J Med 1986
Jul;81(1):69-72
Walton RT, Harvey EL, Dovey S, Freemantle N. Computerised advice on drug dosage to
improve prescribing practice (Protocol for a Cochrane review), In: The Cochrane Library,
Issue 1, 1999. Oxford: Update Software
Murchie CJ, Kenny GN. Comparison among manual, computer-assisted, and closed-loop
control of blood pressure after cardiac surgery. J Cardiothorac Anesth 1989 Feb;3(1):16-9
Competing interests:
No competing interests
22 April 1999
Gunther Eysenbach
Researcher
University of Heidelberg, Dept. of Clinical Social Medicine
Rapid Response:
Moving targets and a insufficient database search in a systematic review?
The "systematic" review of Walton R et al. [1] appears to disregard a
considerable body of knowledge, including controlled trials concerning insulin dosage
recommendation systems [2-8], and illustrates the problems a systematic review may run
into if the research question is not well enough defined in advance.
First, the review states the aim to "identify all comparative studies in which
computers were used to help determine the most appropriate drug dose" and in the
original research protocol authors state to include all studies where participants are
"patients receiving drug therapy based on advice from a computer" [9]. Thus,
according to their own protocol, authors would clearly have to include studies where
advice is given directly to patients or to nurses, such as in some software applications
for diabetes management [2-8]. In the abstract of their review however, authors speak of
"computers giving advice to clinicians" [1] (apparently excluding advice
to patients and direct computer-assisted drug administration), but in the article body
they broaden this concept again to include studies in which computers administer drugs
directly to the patient (but inconsequentially not to those where drug advice is
given to the patient). One would at least have expected some discussion for the reason to
exclude studies where advice is given to the patient, especially in the light of the fact
that insulin treatment in diabetes (having clear physiological parameters) should be a
very nice model to demonstrate the power of computerised drug advice and also because
contamination effects (the administering clinician treating both patients from control and
intervention groups learns from the computer advice of the intervention group) cannot
occur in these studies.
Secondly, the authors used an insufficient search strategy, which may have led to
non-retrieval of studies (e.g [10]) and potential bias. The use of "prescr*" as
a search term for the broad concept of drugs (including medication administered by
computer-controlled infusion pumps) is a major flaw, leading to non-retrieval of at least
261200 MEDLINE articles which deal with pharmaceutical preparations but do not use the
word stem "prescr*". Equally insensitive is the usage of the search term
comput*, (for example, at least 9967 MEDLINE articles use the word "software" or
"expert system" without using " comput*"). To rely on the MeSH term
"Computer-Assisted Decision Making" alone (which was introduced as late as 1987)
is a rather insensitive approach for this broad research question, again especially in the
context of computer-controlled pumps.
Thirdly, the authors claimed to include other research designs such as interrupted time
series, but confusingly this is not reflected in the search strategy "randomised
controlled trial OR random allocation OR double blind method", which appears to
target primarily randomised controlled trials.
In summary, the authors seem to have pursued a moving target, originally having in mind
computerised advice to improve "prescribing practice" (as reflected in the title
of the original research protocol [9]) and subsequently broadening this concept to
computer support for determining drug dose including computer-controlled pumps (which is a
much broader and different concept). Their aim of this review was to determine
"whether there is clear evidence that computerised advice on drug dosage is
beneficial"[9], therefore evidence regarding computer-advice given directly to
patients should have been included in this review (even if this group of studies would
perhaps have required subgroup analysis).
These problems should illustrate the importance of defining a clear search strategy in
advance (which has not been done in the protocol of this particular systematic review [9])
and the importance of consulting an experienced information broker before conducting a
literature search.
Dr Gunther Eysenbach
Researcher, Cochrane Skin Group
University of Heidelberg
Dept. Of Clinical Social Medicine
Unit for Cybermedicine
Bergheimer Str. 58
69115 Heidelberg
Germany
Email: ey@yi.com
References
systematic review and meta-analysis. BMJ 1999 Apr 10;318(7189):984-990
comparing insulin-dosage adjustment by manual and computer algorithms. Diabetes Care 1990
Oct;13(10):1080-4
Regelsystems fuer die Pharmakotherapie mit Insulin [Analytic design and clinical
application of an intelligent control system for pharmacotherapy with insulin--2]. Biomed
Tech (Berl) 1996 Mar;41(3):42-53
computerized diabetes management systems: comparison with the log-book method. Diabetes
Res 1992 Mar;19(3):97-105
Controlled study on the use of hand-held insulin dosage computers enabling conversion to
and optimizing of meal-related insulin therapy regimens. Life Support Syst 1985;3 Suppl
1:561-7
Life Support Syst 1985;3 Suppl 1:568-72
for decision support by multiple regimen insulin dose adjustment. Comput Methods Programs
Biomed 1996 Jan;49(1):105-15
delivery in patients with type I diabetes beginning pump therapy. Am J Med 1986
Jul;81(1):69-72
improve prescribing practice (Protocol for a Cochrane review), In: The Cochrane Library,
Issue 1, 1999. Oxford: Update Software
control of blood pressure after cardiac surgery. J Cardiothorac Anesth 1989 Feb;3(1):16-9
Competing interests: No competing interests