DEVELOPMENT AND VALIDATION OF DECISION TREE MODELS FOR MATCHING CLIENTS TO ADDICTION TREATMENT AND CARE


G.C.M. Kersten, G.M. Schippers, T.G. Broekman, J.M. van Rijswijck & J.P.A. Joosten


University of Nijmegen Research Group on Addictive Behaviors


Bureau Bêta Nijmegen

SUMMARY

The complete report of the project is available in Dutch

The accurate matching of addiction treatment and care possibilities to a particular individual recently has become more important. This is the result of the increasing differentiation of the treatment facilities within a single organization, the increasing complexity of the problems of the clients with addictive behaviors, and the increasing knowledge of the effectiveness of different treatment modalities. This situation calls for a thorough evaluation of the matching and referral practices in the Dutch treatment of addiction. Evaluation can only be undertaken when these practices are explicit, however, and this is currently not the case: The matching and referral rules for the treatment of addiction have simply not been well described. The rules are often invisible, impossible to evaluate, difficult to systematically improve, and largely nontransferable. The goals of the present research were therefore as follows.

  1. To develop and validate models for client-treatment matching and referral in a few locations representing the addiction-treatment practices of the Netherlands.
  2. To evaluate the matching and referral practices on the basis of scientific research.
  3. To promote self-evaluation and possible change in the matching and referral practices used in the treatment of addiction.

The research questions were as follows.

  1. What theoretical and empirical knowledge might contribute to the improvement of the matching and referral of clients with addiction problems, and what guidelines can be derived from this knowledge?
    What are the rules that appear to underlie the matching and referral decisions in a variety of addiction-treatment centers, and what models can be created on the basis of these rules?
  2. To what extent do the formulated models correspond to the actual practice in these treatment centers, and what factors appear to be responsible for potential deviations?
  3. To what extent is our empirical knowledge of matching reflected in the models of actual matching practice?

The client-treatment matching is conceived in the present research as a decision leading to the appropriate form of treatment. A matching decision consists of two steps. (1) The collection of client information, and (2) the application of particular rules to this information. We then introduce a distinction between two types of client characteristics that can play a role in client-treatment matching: namely, treatment-dependent and treatment-independent client characteristics. Treatment-dependent characteristics concern the perspective of the client and are the product of existing treatment possibilities and/or the interaction between the client and the therapist (for example, the particular motives and treatment desires of the client and also previous treatment experience). Treatment-independent characteristics do not concern either the provision of treatment or the therapist. Treatment-independent characteristics are related to the nature of the problem, the characteristics of the individual client, and the social circumstances of the client (for example, addiction severity, psychiatric diagnosis, and living situation).

The conclusion of the literature review undertaken with regard to the results of prediction and matching studies and existing matching and referral models is that relatively few 'hard' empirical facts are available. For a number of client characteristics (and rules), it is clear that they should play a role in client-treatment matching. These include the severity and nature of the psychiatric problem along with the social stability of the client (treatment-independent variables). They also include the particular motivation and treatment desires of the client (treatment-independent variables). The client characteristics and concomitant rules encountered in the literature are divided into six major dimensions: (I) client perspective, (II) previous treatment experience, (III) demographic characteristics, (IV) substance abuse-related characteristics, (V) intra-personal characteristics, and (VI) inter-personal characteristics.

Three addiction-treatment centers - the Bouman House (Boumanhuis) in Rotterdam, the Gelders Center for Addiction Treatment (Gelders Centrum voor Verslavingszorg) and the clinical facility affiliated with the Gelders Center, the Oolgaardt House (Oolgaardthuis) in Arnhem - participated in our study. The Boumanhuis is a big urban treatment center and it consists of two divisions, one alcohol-treatment and one drug-tre atment and every division has several outpatient and inpatient units (nineteen in total). The Gelders Centrum voor Verslavingszorg is a rural outpatient addiction center with locations in five cities with several alcohol- and drug-treatment modalities (such as beha vior thera py, self control training, family therapy, methadone maintenance). The locations Arnhem and Nijmegen participated in the research. The Oolgaard thuis is a clinic with a detoxification and observation unit, and a hier archical therapeu tic community. In these treatment centers in total circa 7000 clients are treated per year.

The matching and referral rules of the three centers were observed, made explicit, and described. In these descriptions, a flow chart has been used to represent each referral point (each of the points where matching decisions are made) and the rules for flowing from one unit to the next. The description and specification occurred in close collaboration with the therapists of the treatment centers. The process was a long, intense, and difficult process of observation, interviewing, description, feedback, and continual adjustment, which took more than two-and-a-half years. The relevant teams were stimulated to improve their matching and referral practices by making them explicit. The result is a series of extensive descriptions of nonprioritized flow rules, that were accepted by the teams as representing acual matching practice. The descriptions of flow rules have been presented in a series of bundles for the different locations. In a total of more than 200 pages, the rules for 44 referral points and more than 100 different referral possibilities (varying from general hospital, several psychiatric inpatient and outpatient modalities to methadone maintenance, group therapy, family therapy, counselling, rehabilitation programs etc.) are described. In the description of the matching and referral rules, a number of concepts (for example, addiction severity, psychiatric problems) appeared to be insufficiently operationalized. This is in part because objective diagnostic instrumentation was either lacking or not in use. The descriptions of the matching and referral practices were also so bulky, poorly organized, and redundant that they simply could not be evaluated.

The model chosen to facilitate the efficient representation and evaluation of the matching and referral practices in addiction treatment was the decision tree. The descriptions of flow rules were transformed into decision trees for the eight largest and most representative referral points in the three addiction-treatment centers: the central screening point for drug addiction, the outpatient observation program for drug addiction, and the inpatient observation program for drug addiction from the Bouman House drug division; the Consultation bureau for Alcohol addiction (CAD) Rotterdam and two alcohol clinics (House ter Schie en Boerhaave Clinic) from the Bouman House alcohol division; the Consultation Bureau for Alcohol and Drugs (CAD) Nijmegen; and the Oolgaardt House. The decision trees were constructed in such a manner that with a minimum of client characteristics and rules an unambiguous indicated treatment alternative could be established. This implied a considerable reduction of the number of necessary rules. To determine the validity of the decision trees, the correspondence between the matching decisions delivered by the decision trees and the matching decisions made in actual practice had to be evaluated. Such validation is above all needed to verify that our decision trees actually fit matching and referral practice or, put differently, that actual practice can be represented in such a manner.

For validation, those client characteristics that occur in the decision trees must first be reliably measured. For this purpose the Assessment Instrument Matching variables (AIM) (Beoordelings-Instrument Cliëntkenmerken, BIC) is developed. The AIM consists of 40 items with regard to client characteristics from the decision trees in the following areas: substance abuse, previous treatment experience, social circumstances of the client, psychiatric and psycho-social problems, and treatment desires of the client.

The reliability and utility of the Asessment Instrument were then studied. Eighteen therapists from nine different Dutch addiction-treatment centers each evaluated 20 (of a total of 60) clients with the aid of the AIM. The clients were recruited from the three centers participating in this research and were interviewed by the researchers. The interviews were recorded on video, and every client was then evaluated by six therapists. In order to determine the inter-rater reliability, kappas were calculated. The inter-rater reliability of 12 items was substantial (.60 < k <= .80), of 11 items moderate (.40 < k <= .60), of 9 items fair (.20 < k <= .40), and 9 items where judged to be insufficent reliable. The most important explanation for the insufficient or fair degree of reliability was commonly the involvement of difficult to operationalize concepts like crisis, psychiatric problems, personality disorders, intro spec tive capacity and fami ly problems .

The utility of the AIM was studied by asking the raters whether they could make a matching decision on the basis of the information from the instrument and the interview or not. In more than 80% of the cases, this question was answered positively. Those instances where the therapists reported not being able to make a matching decision on the basis of the collected information appeared to be largely the result of the occurrence of seven particularly 'difficult' cases in the sample. The results of these analyses suggest that the AIM provides enough information for making a matching decision. It is concluded that the AIM is an economic and pragmatic instrument for client-treatment matching on condition that unreliable items are better operationalized.

Keeping in mind those client characteristics that were difficult to establish and thus less reliable (the crisis items were particularly problematic because they occur high at the 'top' of the decision trees), the validity of the decision trees was next determined. Validation involves a comparison of the matching decisions produced in actual practice with those produced according to the models. The validation research was set up as follows. During a period of one-half to three-quarters of a year at eight key referral points in the three aforementioned addiction-treatment centers, the client characteristics for a particular client were assessed with the aid of the AIM whenever a matching decision was made. The therapists involved in this assessment also recorded their own matching decision and actual referral behavior for the relevant client. With the aid of a computer algorithm specially developed for this purpose, a matching decision on the basis of the client characteristics for each observed client was made in keeping with the relevant decision-tree model. A total of circa 750 client decisions was observed.

The deviations of the matching decisions according to the model from those according to actual practice were analyzed further. There appeared to be two types of causes for deviations: (1) Reliability-related causes; these are deviations between model and practice that are primary caused by ill defined client variables or unclear treatment options of the decision tree models. These observations are left out of consideration in further (validity) analysis. (2) Validity-related causes; these are deviations between model and practice that refer to the validity of the decision trees. There are two kinds of validity-related causes, namely (a) construction inaccuracies in the decision trees (for instance are there recent changes in matching practice that are not yet represented by the decision tree), (b) unclear or inconsistent matching rules. The first kind of deviations could be 'solved' by making the appropriate adjustments in the decision trees. The second kind of deviations were taken as invalidations. That is, the possibility of representing the matching decision within the relevant model is open to question. The percentage of invalidating deviations varied per decision tree and ranged between 3% and 35%: The central screening point for drug addiction 7%, the outpatient observation program for drug addiction 21%, the inpatient observation program for drug addiction 33%; the CAD for alcohol addiction Rotterdam 18%, clinic House ter Schie 19%, and the Boerhaave Clinic 3%; the CAD Nijmegen 35%, and the Oolgaardt House 20%. The percentage of invalidating deviations was in part a product of the chance of error, which varied per decision tree. To the degree that the decision tree contained more decision points and possibilities for referral, the chances of error increased. Similarly, the involvement of a greater number of therapists in the matching meant a greater chance of error. Such an increased chance of error partially explained the relatively high percentage of deviations for the CAD in Nijmegen. The chance of error was also greater when the therapists responsable for the assessments and matching decisions were 'new' and had not, thus, been involved in the development of the descriptions of flow rules. This situation appears to have played a role in the outpatient observation program and the inpatient observation program for drug addiction from the Bouman House. The most important (and intrinsic) threat to the validity of the decision trees consists of ongoing discussions in the treatment teams on a number of the fundamental problems associated with client-treatment matching. These discussions concerned the following points: (1) criteria for addiction severity, (2) matching rules for family therapy, (3) the role of previous treatment-experience in choosing treatment setting (outpa tient versus inpatient), and (4) the role of client's treatment desires.

As in the reliability research, the validation research shows a number of the client characteristics to be insufficiently operationalized. The variables crisis, psychiatric problems, personality disorders characterized by a lack of structure, and introspective capacity appear to be particularly in need of attention. Moreover, a number of the therapeutic alternatives are insufficiently described and distinguished from each other. These include, for example, guidance versus treatment, follow-up care, and outpatient versus ambulatory care. In other words, the better operationalization of both the client characteristics and the treatment alternatives appears to be a prerequisite for formal and verifiable client-treatment matching.

Our conclusion is that the decision trees are reasonably valid. This conclusion does not apply to the decision trees of the CAD Nijmegen and the clinical observation program for drug addiction from the Bouman House. These decision trees accurately represented only about two-thirds of the matching decisions.

In order to more generally evaluate the matching and referral practices, the relative importance of the different client characteristics and thereby the role of the treatment-dependent and treatment-independent client characteristics in matching were determined on the basis of the data from the validation research. The frequency with which the characteristics were found to play a role in the matching decisions and the distinct importance of a particular client characteristic were determined for this purpose. The characteristic of psychiatric problems occurred in all eight of the decision trees, and at least 96% of all the clients were sorted in the decision trees according to this characteristic; this characteristic was found to have influence on the matching decision for less than 6% of the clients, however. Items pertaining to a possible crisis occurred in six of the decision trees and were assessed for 84% of the clients; one can speak of only a limited number of actual cases of crisis, however (somatic crisis 11%, psychiatric crisis 4%, and psycho-social crisis 26%). The characteristic of previous experience with outpatient treatment occurred in four of the decision trees, and 66% of the clients were sorted according to this characteristic; 61% of the clients also had previous outpatient treatment, which suggests that this is a particularly important characteristic for sorting. Less than one-half of the clients were found to be sorted according to any one of the remaining client characteristics.

When we combine the number of clients with a particular characteristic with the number of clients actually sorted according to that characteristic, the order of importance appears to be: use of opiates at this moment, addiction severity, and client desire for methadone (for drug addicted); addiction severity, family problems, and introspective capacity (for alcohol addicted).

The different client characteristics (or items from the AIM) were grouped according to the six major dimensions found in the literature. The frequency with which the characteristics were found to play a role in the matching decisions was determined for the major dimensions. The intra-personal characteristics were found to play the (quantitatively) most important role followed by the inter-personal and substance abuse-related characteristics (with almost equivalent roles), client perspective, previous treatment experience, and demographic characteristics (which were found to hardly play role). The percentages of treatment-dependent and treatment-independent characteristics used in the decision trees to sort clients were 22% (dependent) and 88% (independent).

On the basis of the rank ordering of the client characteristics and our knowledge of the decision trees, a decision hierarchy was constructed to represent the most important client characteristics and treatment dimensions. This hierarchy makes decisions at three different levels: (1) addiction treatment versus psychological/health care, (2) outpatient versus part-time versus inpatient treatment, and (3) the choice of treatment modality. The second level appears to be the most important with regard to the number of clients sorted and capacity to distinguish. A number of decisions at the third level follows - in particular: non-directive versus directive treatment and family versus individual treatment. From this decision hierarchy, the relative contributions of treatment-dependent and treatment-independent client characteristics also become clear in terms of the degree to which these characteristics differentiate between treatment options. The role of the treatment desires of the client, for example, was found to be particularly important at the second and third levels where a large investment by the client (for inpatient treatment) may be expected or a specific treatment motivation (for group treatment, family treatment or treatment in a Hierarchical Therapeutic Community).

The matching and referral practices isolated in this research were next evaluated in light of the existing literature. The question was whether or not the client characteristics and matching rules that have stood out as important in previous empirical research could be detected in the decision trees. The conclusion of this evaluation is that client-treatment matching in the three treatment centers is at places not specific or refined enough. This situation appears to be caused by less than explicit assessment of a number of client characteristics and in particular the psychiatric problems, addiction model entertained by the client, and stage of change that the client is in. The therapeutic alternatives are also not sufficiently differentiated with respect to the aforementioned client characteristics. Those treatment modalities directed at specific psychiatric (DSM-IV Axis I) problems (for example, anxiety disorders, mood disorders) were simply not encountered in any of the decision trees. Those treatment modalities directed at the addiction model of the client and the client's stage of change, such as minimal interventions, self-control procedures, self-help books, and motivation enhancement techniques appear to be lacking in most decision trees.

In general, the matching and referral practices in addiction treatment and care in the Netherlands can, with some effort, be made explicit using relatively simple decision- tree models. The nature of the decision trees corresponds reasonably well to empirical knowledge although further differentiation on the basis of this information may be desired at particular points.

Bureau Bêta Home page