Developing rules and LLM -based systems to define the words of fine -grained categories

Connection table

Summary and 1st Introduction

2 data

2.1 Data Resources

2.2 SS and SI categories

3 Methods

3.1 Dictionary creation and expansion

3.2 Additional Explanations

3.3 System Description

4 results

4.1 Demography and 4.2 System Performance

5 Discussion

5.1 LIMITATIONS

6 CONCLUSION, Repetibility, Financing, Acknowledgments, Author Contributions and References

Complement

Social support and social insulation explanations in clinical notes

Other controlled models

3.3 System Description

In clinical notes, we have developed rules and LLM -based systems to describe the words of fine -grained categories. The rules were then used as a fine grain classification at the asset level grade levels and thinly grain as indicated in section 3.2. An architecture of NLP systems is given in Figure 2.

3.3.1 Rule Based System

As mentioned above, a major advantage of RBS is full transparency of how classification decisions are made. We applied the system using open source Spacy Matcher § [41]. In addition, we have compiled a list of excluding keywords (see additional table s4) to improve the rules.

3.3.2 Controlled Models

By expanding in the published literature, we first started to implement support vector machines (SVMS) and bidirectional encoder representations from the Transformers (Bert) -based models in WCM to identify fine -grained categories. However, these models were inappropriate due to a small number of SS/SI in the corpus (see additional material and table s6).

3.3.3 Big Language Models (LLMS)

We have developed a semi-automatic method to define SS and SI, using an open source advanced fundetuned LLM “Fine-tuned Language Net-T5 (Flan-T5)” [42, 43]. We used the Flan-T5 in the style of “question and answer” to make sentences from clinical texts by talking about SS and SI subcategories. A separate fine -tuned model was created for each of the fine -grained categories.

Model selection: The T5 is used for other classification tasks in clinical notes and does not require labeled training data that uses the FLAN FLAN version of T5, using a chain of thought (cot). [43]. Five types of Flan-T5 are available according to the number of model parameters. Guevara et al. [32] FLAN-T5-XL has observed that smaller models (Flan-T5-L, Flan-T5-T5-T5-Tmall) performed better without a significant recovery with larger FLAN-T5-XXL). Thus, we selected FLAN-T5-XL for our experiments.

ZERO SHOOT: Considering that LLMs follow the instructions and are trained in large amounts of data, they do not necessarily require labeled training data. This “zero jumping” approach was carried out by providing model teaching, context, a question and possible choice (‘yes,’ ‘no’ or ‘relevant’). Table 1 is given an example. The ‘No’ option was selected for disappearing contexts and the subcategory was selected for ‘non -relevant’ for those who are not related to the question.

Fine adjustment: Since the instructed FLAN-T5-XL (zero jump) are weak F scores (see additional table s8), the models have been improved by synthetic examples that could help the model obtain information about specific SS or SI subcategories. Approximately 50 (yes), 50 (no) and 50 (non -relevant) examples have been created for each fine -grained category. Synthetic examples themselves have become a verification set to make the parameters fine -tuning. ChatGPT (with GPT 4.0) ‖ was used to help context examples, but after a few repetitions on the verification set, it was refined by field experts, thus tutoring the closure and exclusion of each sample category. Samples for loneliness are given in Table 1. All fine -tuning samples and questions for each subcategory are given in additional material and Table S7. Furthermore, it has been shown to give LLMS to specific step -by -step instructions (“Instruction TUİNG”), reducing hallucinations and increasing performance. [44, 45]. Therefore, we added an instruction as part of the request.

Parameters: Previously, to describe the SDOH categories, FLAN-T5 models used parameter efficiently low-grade adaptation (Lora) fine tuning method [32]. However, more recently influenced adapter was selected by inhibiting and amplifying internal activations (IA3) for better performance [46]. We adjusted the data in the 15-20 period. Fine -tuning parameters can be displayed in our public code ∗.

3.3.4 EVALUATION

All evaluations were made at a grade level for both thin and rough grain categories. To verify NLP systems, sensitivity, remembering and F-SSKORU were subjected to a macro average to provide equal weight to the number of samples. Emotional support examples and no emotional support subcategory were rare in the underlying notes (see additional table s5 for integers) and therefore accuracy could not be evaluated.

§ https://spacy.io/api/matcher

¶ https://huggingFace.co/docs/transformers/model doc/flan-t5

‖HTTPS: //openai.com/blog/chatgpt

∗ https: //github.com/cornellmhilab/social support social insulation