Your same logic could have been used for the H1N1 influenza that killed upwards of 50M people worldwide and 700K+ Americans when our population was only 100M.
Why? Because like the coronavirus, the H1N1 virus was not only extremely virulent, but also transmissible human-to-human before symptoms showed. What’s more, because the symptoms don’t seem too bad in the beginning, fewer people worry about it and seek treatment.
In other words, while the coronavirus only has about a 2–3% mortality rate (the 1918 influenza had a 5% mortality rate in America), due to how it is so easily transmitted before symptoms show, if it infects many millions (as it certainly has the capacity to do), then it would result in a far higher death toll than you and the other naysayers seem to think. This is why, while scary diseases like Ebola get all the press, the ones more likely to directly affect you and your family are influenzas and coronaviruses just like this one.
This is also why it’s not wise to be so quick to dismiss the media’s efforts to promulgate the warnings of the experts who do know what they’re talking about.
Lastly, while yes, modern medicine and social awareness is going to help cut the mortality rate of the coronavirus, our modern transportation-linked world ensures that the coronavirus is traveling farther and faster than ever before. In other words, yes, you should be deeply concerned.