1. A brief review on benchmarking for large language models evaluation in healthcareLeona Cilar Budler, Hongyu Chen, Aokun Chen, Maxim Topaz, Wilson Tam, Jiang Bian, Gregor Štiglic, 2025, review article Abstract: This paper reviews benchmarking methods for evaluating large language models (LLMs) in healthcare settings. It highlights the importance of rigorous benchmarking to ensure LLMs' safety, accuracy, and effectiveness in clinical applications. The review also discusses the challenges of developing standardized benchmarks and metrics tailored to healthcare-specific tasks such as medical text generation, disease diagnosis, and patient management. Ethical considerations, including privacy, data security, and bias, are also addressed, underscoring the need for multidisciplinary collaboration to establish robust benchmarking frameworks that facilitate LLMs' reliable and ethical use in healthcare. Evaluation of LLMs remains challenging due to the lack of standardized healthcare-specific benchmarks and comprehensive datasets. Key concerns include patient safety, data privacy, model bias, and better explainability, all of which impact the overall trustworthiness of LLMs in clinical settings.
Keywords: artificial intelligence, benchmarking, chatbots, healthcare, large language models, natural language processing
Published in DKUM: 12.05.2025; Views: 0; Downloads: 0
 Link to file |
2. Analysis and synthesis of theoretical and practical implications of case management model and notationMateja Bule, Gregor Polančič, 2025, original scientific article Abstract: Case Management Model and Notation (CMMN) is a graphical notation used to model less predictable, highly flexible processes that may behave differently in each instance. It uses an event-centred approach and expands on what can be modelled with procedural modelling notations. Nearly a decade since the occurrence of CMMN, its practical use is questionable. We performed this research to identify possible reasons for this and to classify the potential advantages and disadvantages of CMMN. With the aforementioned objectives, we conducted a systematic literature review, which provided a broad insight into the state of the investigated object along with techniques for analysing qualitative data, coding, and successive approximation. From an initial set of 942 articles, 43 remain relevant. The results of the analysis and synthesis of the obtained data from relevant articles were generalised codes, which were used to explicitly answer the research questions. The results indicate that CMMN has good foundations in the declarative modelling approach and within the Case Management paradigm. Nevertheless, some issues were identified with the notation and elements of CMMN and with its complement—Business Process Model and Notation (BPMN).
Keywords: CMMN, case management, BPMN, declarative modelling, flexibility, visual language
Published in DKUM: 23.04.2025; Views: 0; Downloads: 0
 Full text (1,00 MB) |
3. On the use of morpho-syntactic description tags in neural machine translation with small and large training corporaGregor Donaj, Mirjam Sepesy Maučec, 2022, original scientific article Abstract: With the transition to neural architectures, machine translation achieves very good quality for several resource-rich languages. However, the results are still much worse for languages
with complex morphology, especially if they are low-resource languages. This paper reports the
results of a systematic analysis of adding morphological information into neural machine translation
system training. Translation systems presented and compared in this research exploit morphological
information from corpora in different formats. Some formats join semantic and grammatical information and others separate these two types of information. Semantic information is modeled using
lemmas and grammatical information using Morpho-Syntactic Description (MSD) tags. Experiments
were performed on corpora of different sizes for the English–Slovene language pair. The conclusions
were drawn for a domain-specific translation system and for a translation system for the general
domain. With MSD tags, we improved the performance by up to 1.40 and 1.68 BLEU points in the
two translation directions. We found that systems with training corpora in different formats improve
the performance differently depending on the translation direction and corpora size.
Keywords: neural machine translation, POS tags, MSD tags, inflected language, data sparsity, corpora size
Published in DKUM: 28.03.2025; Views: 0; Downloads: 5
Full text (448,16 KB)
This document has many files! More... |
4. Evolution of domain-specific modeling language: an example of an industrial case study on an RT-sequencerTomaž Kos, Marjan Mernik, Tomaž Kosar, 2022, original scientific article Abstract: Model-driven engineering is a well-established software development methodology that
uses models to develop applications where the end-users with visual elements model abstractions
from a specific domain. These models are based on domain-specific modeling language (DSML),
which is particular to the problem domain. During DSML use, new ideas emerge and DSMLs evolve.
However, reports on DSML evolution are rare. This study presents a new DSML called RT-Sequencer
that evolved from our DSML Sequencer to support, in addition to the Data Acquisition domain,
also a new domain—Real-Time Control (RTC) systems. The process of defining models with a new
language RT-Sequencer has changed in a way that new end-users were introduced—advanced endusers, which use general-purpose language (GPL) and advanced programming concepts to define
modeling environments for the RT-Sequencer end-users. More specifically, an industrial experience
with the RT-Sequencer is presented, where DSML was opened for extension so that a GPL code
could be inserted into the model to create new visual blocks for the end-user, and the possibility to
adapt and optimize the execution code for a particular task. Our experience shows the specific case
of DSML evolution supporting another problem domain, and the implementation effort needed to
extend domain-specific modeling language with GPL support.
Keywords: model-driven engineering, domain-specific modeling languages, measurement systems, Real-Time Control systems, data acquisition, language evolution, experience report
Published in DKUM: 27.03.2025; Views: 0; Downloads: 2
Full text (1,70 MB)
This document has many files! More... |
5. Analysis of it solutions to improve the inclusiveness of foreign language speakers : master's thesisNikola Vilar Jordanovski, 2025, master's thesis Abstract: This thesis explores how information technologies can enhance linguistic inclusivity in a
globalized society, where language barriers are increasingly evident due to greater
mobility and cultural interconnectedness. The empirical section compares ten IT
solutions, five offline and five using artificial intelligence, by translating ten common
email messages into five languages. The analysis focuses on translation accuracy,
reliability, and user experience. User surveys provided additional insights into challenges
and desired functionalities for greater inclusivity.
Findings indicate that AI-based solutions like ChatGPT and DeepL achieve better
contextual accuracy. The thesis suggests that future development of these solutions
should emphasize cultural nuances and contextual precision to improve accessibility and
inclusivity for speakers of different languages.
Keywords: inclusiveness, foreign language speakers, IT solutions for inclusiveness, multilingualism
Published in DKUM: 27.03.2025; Views: 0; Downloads: 13
Full text (1,86 MB) |
6. Usefulness of digital language resources in improving native language among adultsSuzana Žilič Fišer, Jani Pavlič, Ines Kožuh, 2022, original scientific article Abstract: Important keys to effective communication are language competences, which can be
supported by using digital language resources. These usually assist the acquisition of a second
language, despite their potential for improving one’s native language. Our study was, thus, aimed at
raising awareness about the possibilities of improving the native language of an adult population
by using digital language resources for the Slovenian language. We conducted workshops, a survey
and, partly, semi-structured interviews with 124 participants. We examined whether the perceived
usefulness and ease of using digital language resources depends on age, education, self-assessed
language proficiency, and experience with language training. The analysis revealed that self-initiative
use of analogue language resources is related positively to using digital ones for seeking information,
improving language use, as well as for study or work. Moreover, self-assessed proficiency in language
was found to affect the perceived ease of using digital language resources. These findings may help
language professionals support developing language skills by using digital language resources and
preserving language in an adult population.
Keywords: digital language resources, native language, language improvement, perceived usefulness, perceived ease of use
Published in DKUM: 27.03.2025; Views: 0; Downloads: 2
Full text (291,60 KB)
This document has many files! More... |
7. |
8. New approach for automated explanation of material phenomena (AA6082) using artificial neural networks and ChatGPTTomaž Goričan, Milan Terčelj, Iztok Peruš, 2024, original scientific article Abstract: Artificial intelligence methods, especially artificial neural networks (ANNs), have increasingly been utilized for the mathematical description of physical phenomena in (metallic) material
processing. Traditional methods often fall short in explaining the complex, real-world data observed
in production. While ANN models, typically functioning as “black boxes”, improve production
efficiency, a deeper understanding of the phenomena, akin to that provided by explicit mathematical
formulas, could enhance this efficiency further. This article proposes a general framework that
leverages ANNs (i.e., Conditional Average Estimator—CAE) to explain predicted results alongside
their graphical presentation, marking a significant improvement over previous approaches and those
relying on expert assessments. Unlike existing Explainable AI (XAI) methods, the proposed framework mimics the standard scientific methodology, utilizing minimal parameters for the mathematical
representation of physical phenomena and their derivatives. Additionally, it analyzes the reliability
and accuracy of the predictions using well-known statistical metrics, transitioning from deterministic
to probabilistic descriptions for better handling of real-world phenomena. The proposed approach
addresses both aleatory and epistemic uncertainties inherent in the data. The concept is demonstrated through the hot extrusion of aluminum alloy 6082, where CAE ANN models and predicts
key parameters, and ChatGPT explains the results, enabling researchers and/or engineers to better
understand the phenomena and outcomes obtained by ANNs.
Keywords: artificial neural networks, automatic explanation, hot extrusion, aluminum alloy, large language models, ChatGPT
Published in DKUM: 27.02.2025; Views: 0; Downloads: 5
Full text (3,18 MB)
This document has many files! More... |
9. Exploring the feasibility of generative AI in persona research : a omparative analysis of large language model-generated and human-crafted personas in obesity researchUrška Smrke, Ana Rehberger, Nejc Plohl, Izidor Mlakar, 2025, original scientific article Abstract: This study investigates the perceptions of Persona descriptions generated using three different large language models (LLMs) and qualitatively developed Personas by an expert panel involved in obesity research. Six different Personas were defined, three from the clinical domain and three from the educational domain. The descriptions of Personas were generated using qualitative methods and the LLMs (i.e., Bard, Llama, and ChatGPT). The perception of the developed Personas was evaluated by experts in the respective fields. The results show that, in general, the perception of Personas did not significantly differ between those generated using LLMs and those qualitatively developed by human experts. This indicates that LLMs have the potential to generate a consistent and valid representation of human stakeholders. The LLM-generated Personas were perceived as believable, relatable, and informative. However, post-hoc comparisons revealed some differences, with descriptions generated using the Bard model being in several Persona descriptions that were evaluated most favorably in terms of empathy, likability, and clarity. This study contributes to the understanding of the potential and challenges of LLM-generated Personas. Although the study focuses on obesity research, it highlights the importance of considering the specific context and the potential issues that researchers should be aware of when using generative AI for generating Personas.
Keywords: user personas, obesity, large language models, value sensitive design, digital health interventions
Published in DKUM: 14.02.2025; Views: 0; Downloads: 5
Full text (812,18 KB) |
10. Strategies for managing time and costs in speech corpus creation : insights from the Slovenian ARTUR corpusDarinka Verdonik, Andreja Bizjak, Andrej Žgank, Mirjam Sepesy Maučec, Mitja Trojar, Jerneja Žganec Gros, Marko Bajec, Iztok Lebar Bajec, Simon Dobrišek, 2024, original scientific article Abstract: Parliamentary debates represent an essential part of democratic discourse and provide insights into various socio-demographic and linguistic phenomena - parliamentary corpora, which contain transcripts of parliamentary debates and extensive metadata, are an important resource for parliamentary discourse analysis and other research areas. This paper presents the Slovenian parliamentary corpus siParl, the latest version of which contains transcripts of plenary sessions and other legislative bodies of the Assembly of the Republic of Slovenia from 1990 to 2022, comprising more than 1 million speeches and 210 million words. We outline the development history of the corpus and also mention other initiatives that have been influenced by siParl (such as the Parla-CLARIN encoding and the ParlaMint corpora of European parliaments), present the corpus creation process, ranging from the initial data collection to the structural development and encoding of the corpus, and given the growing influence of the ParlaMint corpora, compare siParl with the Slovenian ParlaMint-SI corpus. Finally, we discuss updates for the next version as well as the long-term development and enrichment of the siParl corpus.
Keywords: recording speech, transcribing speech, transcription guidelines, Less-resourced language
Published in DKUM: 04.02.2025; Views: 0; Downloads: 8
Full text (1,09 MB)
This document has many files! More... |
