Scarlett Johansson was pregnant during filming, so many of her scenes were scheduled early in filming before she began to show. To help hide her pregnancy later in filming, three stunt doubles were hired. This caused a lot of confusion amongst the other actors since, according to them, all of the stunt women looked very similar to Johansson. Chris Evans stated that it got to the point where he would say hello and start a conversation with one of them only to realize midway that the person he was talking to wasn't Johansson. Ultimately, some scenes used CGI to hide Johansson's belly.
Because Ultron was eight to nine feet tall, the five-foot ten-inch James Spader had to wear an antenna-like contraption made out of thick wire with two red balls attached to the top that went up his entire back and three feet above his head. This was done so that the cast members that shared scenes with him would have a reference point for where his eyes would be. The two red balls represented the placement of Ultron's eyes. Elizabeth Olsen stated that this was actually distracting because Spader would be giving an intense performance, and, out of instinct, she would look at him rather than the balls representing his eyes. Much to everyone's amusement, whenever this happened, Aaron Taylor-Johnson would yell "Red balls! Look at his balls, Lizzie!" at her in order to get her to look in the right direction.
Aaron Taylor-Johnson was asked to lose weight (muscle mass in particular) because he looked too strong to portray a runner, who is usually trim.
James Spader was Joss Whedon's only choice for Ultron because of his "hypnotic voice that can be eerily calm and compelling while also being very human and humorous." Kevin Feige confirmed that Spader was the only actor Marvel spoke with for the role.
On James Spader's first day on-set, the cast was so impressed by his performance that they applauded and cheered for him after his first take.