Pattern formation in a nonlocal mathematical model for the multiple roles of the TGF-β pathway in tumour dynamics

The growth and invasion of cancer cells are very complex processes, which can be regulated by the cross-talk between various signalling pathways, or by single signalling pathways that can control multiple aspects of cell behaviour. TGF- β is one of the most investigated signalling pathways in oncology, since it can regulate multiple aspects of cell behaviour: cell proliferation and apoptosis, cellcell adhesion and epithelial-to-mesenchimal transition via loss of cell adhesion. In this study, we use a mathematical modelling approach to investigate the complex roles of TGF-β signalling pathways on the inhibition and growth of tumours, as well as on the epithelial-to-mesenchimal transition involved in the metastasis of tumour cells. We show that the nonlocal mathematical model derived here to describe repulsive and adhesive cell-cell interactions can explain the formation of new tumour cell aggregations at positions in space that are further away from the main aggregation. Moreover, we show that the increase in cell-cell adhesion leads to fewer but larger aggregations, and the increase in TGF-β molecules – whose late-stage effect is to decrease cell adhesion – leads to many small cellular aggregations. Finally, we perform a sensitivity analysis on some parameters associated with TGF-β dynamics, and use it to investigate the relation between the tumour size and its metastatic spread.