Cancer is the second leading cause of death in the United States, claiming the lives of more than half a million people every year. Cancer is aggressively treated with surgery, chemotherapy, and radiotherapy. The primary focus of this thesis is to assist clinicians with hypothesis generation to design novel radiotherapy and chemoradiotherapy fractionation schemes that can improve the results of current clinical practices. We find solutions for some important questions in radiotherapy and chemotherapy fractionation problem. Chapter 2 extends the model developed in the literature to consider radiotherapy fractionated schedules in glioblastomas to best minimize toxicity arising in early- and late-responding tissues. To this end, we decomposed the problem into two separate solvable optimization tasks: optimal radiation schedule or the amount of radiation dose per fraction and optimization of the amount of time that passes between radiation doses. Chapter 3 proposes a method for determining the optimal fractionation in the presence of uncertainties in model parameters. We formulated our problem as a conservative model using robust optimization and a risk adjusted probabilistic formulation. A variable transformation and branch and bound algorithm is implemented to find the optimal regimen. Chapter 4 considers the radiotherapy fractionation problem with a new objective: minimizing production of metastatic cancer cells while keeping normal tissue damage below an acceptable level. A dynamic programming (DP) framework is utilized to determine the optimal fractionation scheme. In Chapter 5, we introduce a mathematical model to obtain optimal drug and radiation protocols in a chemoradiotherapy scheduling problem with two objectives: minimizing metastatic cancer cell populations at multiple potential sites and maintaining a minimum level of control to the primary tumor site. We derive closed-form expressions for optimal chemotherapy fractionation regimens in some special cases. A DP framework is used to determine the optimal radiotherapy fractionation regimen. Using discretization approach, the exact solution of the resulting DP algorithm is computationally intractable. We design efficient DP data structure and use some structural properties of the optimal solution to reduce the complexity of the resulting DP algorithm. In all chapters, we performed substantial numerical experiments to validate our results.