The 10-man reigning champions were in deep trouble when Kroos stepped up to steer in his dramatic winner in the final minute of time added on at Sochi's Fisht Stadium.
Before that, a Kroos error had led to Ola Toivonen giving Sweden a 32nd-minute lead and even a Marco Reus equaliser early in the second half appeared to be only a temporary stay of execution for Joachim Loew's men. But the rescue act from Kroos, who had endured another difficult night in the German midfield, is a huge lift to a side who saw Jerome Boateng sent off late in the game.
They now have qualification for the last 16 back in their own hands ahead of their final Group F game against South Korea on Wednesday. There is still no guarantee that they will avoid the fate of the last two World Cup holders, with Spain four years ago and Italy in 2010 both being knocked out in the group stage.
Not since 1938 have Germany been eliminated from the World Cup in the first round. However, this result, and the manner of it, will provide an almighty boost to the Germans and is a hammer blow to Sweden, who will face off with Mexico in their last match with all still to play for.
Mexico's 2-1 win over South Korea earlier on Saturday had ensured that Germany would be out in the event of a defeat, with a draw little better.
Germany coach Joachim Loew wielded the axe following the shambolic 1-0 loss at the hands of Mexico with which they began their defence of the trophy, with the much-criticised Mesut Ozil and Sami Khedira dropped while Mats Hummels
missed out due to injury.
Marco Reus, Sebastian Rudy and Antonio Ruediger were among those who came into the team. But just like against Mexico, Germany's usual composure was not there.
Sweden denied penalty
Sweden should have had an early penalty when Marcus Berg was released on goal and, at the point of shooting, was fouled by Boateng. Berg's effort was saved by Manuel Neuer. Boateng played the man and not the ball, but there was no penalty given and no recourse to the Video Assistant Referee.
It was a warning for the Germans, who were then dealt a body blow — literally — when Rudy was left with a bloodied nose after receiving a stray boot in the face.
Ilkay Gundogan replaced him, coming in alongside Kroos, but it was the Real Madrid man who was at fault as Sweden went in front just a minute later. His pass was pounced upon by the Swedes, who sprung forward, Viktor Claesson picking out Toivonen to control on his chest and lift the ball over Neuer with the aid of a slight deflection.
Loew sent on Mario Gomez in place of Julian Draxler at half-time, and it took just three minutes of the second half for the equaliser to arrive.
Timo Werner's low centre from the left was behind Gomez, who got a touch but not enough to put off Reus from bundling it in. They then set off in search of the winner, with Gomez twice coming close and fellow substitute Julian Brandt smacking a post late on.
Boateng had already walked by then, seeing a second yellow in the 82nd minute for a foul on Berg.
Hope looked to be gone, but then Kroos exchanged passes with Reus at a free-kick on the left edge of the box and sent a sweet strike into the far corner.