My husband M35 wants to bring some of his family to online with us. I F32 really don’t want to, how do I not ruin my marriage?

So my husband M35 is a very sweet person, his dad abandoned the family when he was 7 years old, so he stayed with his mom and little sister, he grew up believing he had to be in charge of them because he was the “man of the house”. So nowadays, we on-line in California and his family lives in Latin America, he sends money to her mom (she doesn’t work) and her sister is a single mom with two kids, he also helps her sister economically speaking. Recently he mentioned to me he would like to do all the paper work and bring them to online with us (mom, sister and two nephews) I don’t agree and I don’t want to, I told him to go ahead if he has the money to provide housing and all the necessities for them while they adjust and so on, but that I was not going to work extra for our plans and goals to support a whole other family I didn’t choose to have. One of the reasons I don’t support that is because I don’t believe he has to be responsible for them anymore. His sister is a grown up woman who decided to have 2 kids. And I’m not willing to give the peaceful life we have (we have no kids) and even go thru financial difficulties because my husband feels he’s still responsible for his family. Am I being too selfish? How do I tell him that is not a good idea I already said I’m not ok with it, and told him I was afraid that could put our relationship in danger.