The idea that God is infinite is a relatively late one. Up until the 4th century, no one really thought that God was infinite, partly because they didn't look very kindly on the idea of a being with no limits and therefore no real shape – it made God seem a big like a big amoeba, and nobody wants to worship that.
Gregory of Nyssa was really the first person to argue that God was infinite. He was one of the Cappadocian Fathers, who were Gregory, his older brother Basil, and their mate Gregory of Nazianzus. They were responsible for formulating the doctrine of the Trinity as we have it, really, and so are quite important, except Gregory doesn't get quite as much kudos, on account of his being a bit theologically dodgy and suggesting that maybe God saves everyone in the end (that's called universalism, folks, and maybe we'll talk more about that some other time). Most of the most fun theologians were accused of heresy at some point, though, so don't let that put you off, not least because Gregory Nyssen was definitely the most fun Cappadocian. Anyway, infinity. Gregory's reasoning went like this: if God isn't infinite, he must have limits, or edges. The only possible limit of virtue is evil, but God can't have an opposite, so he must be unlimited and infinite.
The fun thing about this is that it means that there's no end to God, and therefore there doesn't need to be any end to our approach to him. It's a bit like that thing where the bigger the bowl you put your goldfish in, the bigger they'll get, and we're like goldfish in an infinite bowl. We follow God, we get to know him better, but there's still more to know, and so our pursuit of God is both always successful and never complete.
I like this for two reasons:
1) It makes the prospect of eternity a whole lot more fun. “Heaven” or wherever we end up won't be sitting about on clouds getting bored, but will be all about moving towards God, learning more things, understanding him better, becoming more ourselves, loving more fully. More like the Last Battle, where they just go onwards and upwards and onwards and upwards – there are always new things to discover, and God will never be one of those friends who you meet for a coffee and then run out of things to say after you've caught up on what you've been doing since you last met up.
2) It means that we don't get it yet. Sure, we know God - we really know him and what we know is true – but what we know is still only a tiny fraction of everything there is. If only we could all get our heads round this properly, we would all (and I include myself in this), especially us Christians, who tend to equate knowing something of God with knowing more than anyone else, be a lot more fun to be around, and would probably screw things up a lot less often.