The schoolboy, who cannot be identified, told Preston Crown Court that he knew one of the two defendants, child A, very well and knew child B 'a little bit'.
He saw the three of them walking in Church Road West, in Walton, Liverpool, on the day of James's disappearance from a shopping precinct in Bootle. The boy, identified only as E, said both the older boys had hold of James's hand.
Mr Richard Henriques QC, for the prosecution, asked E: 'Did you notice anything about James?' The boy replied that James had a bump above his left eye and it looked as though he had been crying.
He said he had asked child A how the child had done it and A had replied: 'He fell over at the top (of the street).' He said he had not seen the little boy before and asked who he was. Child A had replied he was child B's brother.
He said he then asked where they were going with the child and they said they were going home to child B's. Asked whether he had said anything to them, E said: 'Yeah. If you don't take him home, I'll batter you.'
A second 12-year-old boy, identified only as W, said he recognised two of the three boys as pupils at his former junior school and he knew one of them, child A, by name.
'They were with a young boy, about two or three. I hadn't seen him before. I saw a lump on his head. It was grazed above his eye. His hair was covering it but I could see it - he was crying.
'I said, 'Why is he crying?' and a lump was showed to me by one of the boys.
'I looked at the lump. I said, 'Where are you taking him?' One of the boys said, 'I was taking him home.'
'I thought they were taking him home. I had no idea who the little boy was. We just walked away and they just walked a different way.'
The Crown alleges child A and child B, who were aged 10 at the time, stoned and battered two-year-old James to death on a railway track in Walton after abducting him from the shopping precinct on 12 February this year. They deny the charges and the attempted abduction of another two-year-old from the precinct on the same day.
The court was later told how James and his alleged abductors called at a pet shop. Frances Smith, 36, a shop assistant, said one boy remained with the child while the other went to a tank and claimed a fish was dead. She told them to leave.
Another woman, Hazel Robinson, told the court she saw them recklessly crossing County Road, Walton, the child 'far too young to be entrusted in their care'.
Then William Howes told the jury he knew A and A's mother. On 12 February he noticed James, sobbing, standing close to an older child who was talking to another boy standing above, on a bridge.
Mr Howes, 42, told the court the boy closest to James said: 'I'm fed up having my little brother. It's always the same. I have him from school. I'm going to tell me mum I'm not going to mind him no more.'
Mr Howes said he had been affected by the incident. 'I know that, if I'd stopped, I'd have said something.'
The trial continues today.