Let us designate rose as "R", marigold as "M" and dahlia as "D".
One R costs five rupees, one D costs one rupee and 20 M's cost one rupee. The total spend is 100 rupees. Therefore, 5R + D + 0.05M = 100 (because one marigold costs 5 paise or 0.05 rupee).
Besides, R + D + M = 100 (because the total number of flowers is 100).
Solving the two equations, we get 4R - 0.95M = 0.
Therefore, 4R = 0.95M.
Now I have used a trial and error method. You perhaps have a better method. Anyway, I take M = 1 and I get 4R = 0.95 which gives me R = 0.95/4, that is 0.2375 - this implies I get a few petals instead of the whole rose (of course, if the florist agrees to it). This will not do.
Next, I take M = 2 and I get 4R = 1.9. Therefore, R = 1.9/4, that is 0.475. This will also not do.
I continue in this manner till I reach M = 80. Now I get a nice round figure. When M = 80, I get 4R = 76. So R = 19.
That is it. Atul has decided to buy 80 marigolds and 19 roses. Now for the dahlias. R + D + M = 100, so 19 + D + 80 = 100, so D = 1. Atul can buy only one dahlia. But never mind, he now has all three types of flowers and he has spent the entire 100 rupees.
Therefore, Atul buys 19 roses for 95 rupees, one dahlia for one rupee and 80 marigolds for four rupees. Atul's mother is happy that her son knows his arithmetic.