A HYBRID APPROACH FOR REVOLUTIONIZING MEDICAL INFORMATION RETRIEVAL USING AI

Abstract

Medical question answering systems face significant challenges in providing accurate, timely, and context-aware responses due to the complexity and vast scope of biomedical information. This paper presents a novel hybrid approach that combines BioBERT, a specialized biomedical language model, with the Qdrant vector database to create an efficient and accurate medical question answering system. Our methodology encompasses three key components: (1) comprehensive data preprocessing using KeyBERT for intelligent tagging of the MedQuAD dataset containing 47,457 medical question-answer pairs, (2) finetuning BioBERT with an innovative negative sampling strategy based on semantic tags, and (3) implementation of efficient semantic search using Qdrant vector database for rapid retrieval of relevant responses. The system evaluation demonstrated clear separation between correctly classified positive and negative samples in cosine similarity scores, with misclassifications occurring primarily near the 0.3 threshold. The resulting chatbot system successfully combines domain-specific language understanding with efficient vector-based retrieval to deliver accurate and contextually relevant medical information. This approach addresses the limitations of existing medical question answering systems by providing a scalable, automated solution that maintains high accuracy while reducing dependence on human experts.