Responding to the tumor microenvironment, functional polymers can serve as preeminent drug carriers for targeted cancer therapy. Stimuli-responsive polymeric drug carriers are reported with diverse anti-tumor effects for various polymer structures. Thus, three pH/redox dual-responsive amphiphilic zwitterionic polymer ‘isomers’ with different locations of pH/redox responsive units were prepared to understand the relationship between polymer structure and anti-tumor effect. Containing poly(ε-caprolactone) (PCL), poly(N,N-diethylaminoethyl methacrylate) (PDEA) and poly(2-methacryloyloxyethyl phosphorylcholine) (PMPC), polymers PCL-ss-P(DEA-r-MPC) (SDRM), PCL-ss-PDEA-b-PMPC (SDBM) and PCL-PDEA-ss-PMPC (DSM) with a precisely controlled structure were constructed and confirmed through NMR, FITR and EA. The formed micellar drug carriers were characterized by their morphology, loading capacity, acid/redox sensitivity, drug release, in vitro cytotoxicity and in vivo antitumor effects. Micelles with uniform spherical morphologies can effectively encapsulate anti-tumor drugs such as DOX. Among these micelles, DSM@DOX displays the most excellent drug encapsulation capacity (13.4%) with neutral surface charge (−1.02 mV) and good stability, and is different from SDRM@DOX with positive charge (+11.1 mV) and SDBM@DOX with poor stability. All micelles respond to acid and reducing environments and present fast drug release at mildly acidic pH and high concentrations of GSH, exhibiting low burst release under the physiological conditions of plasma. There is no significant difference between these micelles in tumor cell cytotoxicity against MCF-7 and 4T1 cells. Internalization of SDRM@DOX and DSM@DOX by the tumor cells is stronger than that of SDBM@DOX. Notably, DSM@DOX has longer blood circulation and more effective accumulation at the tumor site than the other two micelles. As a result, DSM@DOX shows enhanced antitumor efficacy in 4T1 tumor-bearing mice with reduced side toxicities. Overall, structures of the above polymers significantly influence the in vivo antitumor effects of the drug carriers through blood circulation and cellular uptake.