We present the effect of losses and nonlinearity on the behavior of modulation instability (MI) in thulium-doped fiber amplifier (TDFA) operational in S and near-C bands. Based on a derived analytical model, the relative change in maximum integrated gain at the respective perturbation frequency was observed and compared with lossless TDFAs. At an estimated small-signal attenuation of 0.36 (corresponding to small-signal attenuation coefficient α  =  0.01  /  m), a frequency shift of 1.44 THz with a reduction in integrated gain of 45 dB was obtained for an input signal power of 250 mW. For the same set of parameters, the effect of nonlinear and loss factor was calculated and compared with present day state-of-the-art erbium-doped fiber amplifiers. The analysis helped in the selection of optimum values for small-signal gain coefficient (g₀  =  0.21  /  m) and TDFA length (L  =  100  m) to maximize the integrated gain of the perturbations for different input signal powers. The special case pertaining to nonideal conditions with respect to the large amplitude of input perturbations was solved numerically. Dipole relaxation time was estimated (450 fs) from the results and compared with the theoretically calculated value (318 fs) for TDFAs. The results obtained can be used constructively in the selection of optimum parameters and perturbation frequency value for the design of TDFA lasers in which MI acts as an active saturable absorber.