The emerging field of explainable Artificial Intelligence (XAI) aims to bring transparency to today’s powerful but opaque deep learning models. However, the vast majority of current approaches to XAI only provide partial insights and leave interpreting the model’s reasoning to the user. This talk will provide an overview on how established techniques from XAI can be used to successfully understand, debug and improve machine learning models and pipelines. The session will provide an outlook on how recent developments towards concept-based XAI (with a focus on the recent Concept Relevance Propagation (CRP) approach) will lead to more human interpretable explanations and thus enable novel analyses to gain insights into the reasoning of AI.