I'm betting that we are reaching the endpoint. The recent changes are over just a few decades, too slow for genetic causes, so I'm guessing that the reason is higher calorie diets. Now that almost everyone in the 1st world is pretty well fed, there isn't much room for improvement.
The same thing is going on with intelligence. The Flynn effect is sputtering out now that we are rich enough to give all children reasonably good meals and schools. A similar effect is happening with Olympic records. Now that we are close to training athletes optimally, records are not being broken by as large an amount as before.
Once genetic engineering takes hold, all bets are off.

When I was a kid I used to visit Portugal a lot. Last time was in 2001 (and I suppose I wasn't a kid anymore).

In 2001, there was a STARK height difference between adults over 30 or so, and the late adolescence / 20-something crowd. This was within the native Portuguese population, not immigrants from Africa, Ecuador etc.

I think something drastic changed in Portugal during the 1970s...