Purpose: The in vitro clonogenic assay (IVCA) is the mainstay of quantitative radiobiology. Here, we investigate the benefit of a time-resolved IVCA version (trIVCA) to improve the quantification of clonogenic survival and relative biological effectiveness (RBE) by analyzing cell colony growth behavior. Materials & Methods: In the IVCA, clonogenicity classification of cell colonies is performed based on a fixed colony size threshold after incubation. In contrast, using trIVCA, we acquire time-lapse microscopy images during incubation and track the growth of each colony using neural-net-based image segmentation. Attributes of the resulting growth curves are then used as predictors for a decision tree classifier to determine clonogenicity of each colony. The method was applied to three cell lines, each irradiated with 250 kV X-rays in the range 0-8 Gy and carbon ions of high LET (100 keV/µm, dose-averaged) in the range 0-2 Gy. We compared the cell survival curves determined by trIVCA to those from the classical IVCA across different size thresholds and incubation times. Further, we investigated the impact of the assaying method on RBE determination. Results: Size distributions of abortive and clonogenic colonies overlap consistently, rendering perfect separation via size threshold unfeasible at any readout time. This effect is dose-dependent, systematically inflating the steepness and curvature of cell survival curves. Consequently, resulting cell survival estimates show variability between 3% and 105%. This uncertainty propagates into RBE calculation with variability between 8% and 25% at 2 Gy.Determining clonogenicity based on growth curves has an accuracy of 95% on average. Conclusion: The IVCA suffers from substantial uncertainty caused by the overlap of size distributions of delayed abortive and clonogenic colonies. This impairs precise quantification of cell survival and RBE. By considering colony growth over time, our method improves assaying clonogenicity.