During brachytherapy sessions, medical specialists record ultrasound images of prostate cancer patients and segment these images manually. In the process of analyzing patient records, it is a fundamental task to localize the catheters (needles) in the recorded ultrasound images. Due to the signi cant amount of noise in ultrasound images, localizing multiple catheter positions in ultrasound images is more challenging than similar image segmentation tasks for MRI and CT images. The manual segmentation process is very time-consuming and relies on experienced clinicians. Hence, a tool for the automatic localization of catheters in ultrasound images is highly desirable. In the medical eld, deep learning has gained popularity due to its ability to produce highly accurate detection tools. In order to automate the segmentation of ultrasound images in general and the detection of catheters in particular, we utilize a deep neural network-based architecture named U-net. A 5-fold cross-validation method is applied to evaluate the performance of the U-net based model on the limited dataset. After training a deep neural network, we further improve the resulting detection model by incorporating domain information provided by a medical expert. i The second step of our approach is to generate 3D detections from a combination of the 2D detections, using sequences of 2D images for each patient. Through these 3D detections, we obtain an improvement in accuracy compared to the 2D model.