we do live in a finite world
You've inverted the problem. The issue is productive efficiency. If we had perfect efficiency, we'd get output goods and services with zero input costs. Since zero input costs is, essentially, impossible, we always have the ability to improve efficiency.
To be fair, the metrics are somewhat inverted as well. GDP is only a pseudo-metric for measuring productivity, generally in terms of trends, but also changes with the population. GDP per capita over time is a better measure of productive efficiency as it provides the output per citizen. Of course that all depends on the demographics, average hours working, unemployment rate, etc., etc. And GDP doesn't directly measure useful output so much as all output. More disease and health problems can drive up GDP, for instance (unless it takes people out of the workforce).
Ideally you could measure useful output for a fixed input, e.g., per hour of labour worked. That's not bad for efficiency measure, but then you lose the cumulative "national health" component of GDP. That is, if only one person in the whole country is working and is very efficient, the national efficiency would look good even though almost everybody is unemployed and starving. GDP sort of includes an implicit component of employment rate, or at least mixed with effects of population growth and demographic changes.
But really efficiency should be measured based on the input labour hours required to produce a fixed output. Imagine if you could produce all of your monthly needs, including a luxury home, with only 1 hour of work. That is prosperity and quality of life. It is what "wealth" really means. (Money is only a middleman proxy representation. What the money can buy you, and how hard/long you have to work to earn the money, are what really matter.)
So yes, it can be infinite. Cutting input labour in half will double efficiency, but there is always still time left to cut in half again.
As far as debt, if the debt is used to cut the labour required in half, as long as that savings in costs (or alternately, that generated profits, or generated wealth) exceeds the interest on the debt, then it is a net value.
Think of a student loan. You could choose not to go to university, and end up making $X/year. If you take out a student loan, go to university, and as a result of your education you make $Y/year, you are earning $(Y-X)/yr more. If you use that extra money to pay off the loan in, say, 5 or 10 years, then every year after that for the rest of your life will be a net profit of $(Y-X) over and above what you would have earned had you not done it. Getting that loan was easily worth it in that case.
Now imagine instead of paying it off you just pay the interest on it, $i. Now every year starting immediately you have a net profit of $(Y-X-i). So you have an earlier profit, and can perhaps invest that profit at a return rate higher than i. (In fact, this is what I did and stretched the payments for as long as I could.) The problem with doing this is that your life is finite, and you'll still owe the loan at the end from your estate. Maybe that isn't a problem.
But a country doesn't die. It can perpetually keep paying that interest and living on the $(Y-X-i) profit from the beginning and forever. And the loaner makes i% on their investment as well. It's win-win. This may seem to create something from nothing, but the wealth is created by the improved efficiency. That will generally be some sort of technology or division of labour (e.g., comparative advantage) that is more efficient and produces the same for less. The net wealth generated is (Y-X) per year, of which the borrower keeps the (Y-X-i) portion and the lender keeps the i portion.
Does that help?