Spender's childhood, based on this poem, was lonely and he spent much of his time frightened of the boys in his neighborhood.
The majority of the poem describes these rough boys who held Spender down. He describes their knees on his arms and the way they mocked his lisp. They threw mud at him and were never nice to him.
The beginning of the poem, however, makes clear that his parents kept him away from these kids. As a child, he was therefore lonely and scared.
Looking back on his childhood, the poet expresses sadness that these boys never showed him kindness. The final line shows that as a boy he would have forgiven them if they only smiled -- which they never did.
