The clash occurred at about 03:00 local time after a tip-off about the location of a suspected Boko Haram weapons cache, Nigeria's spy agency said.
The State Security Service did not give any details about casualties. A witness told the BBC he saw nine bodies.
However, other witnesses told Reuters news agency the shooting came during an attempt to move squatters.
Six witnesses told Reuters the house was owned by a military man who wanted them to leave his property.
The BBC's Mohammed Kabir Mohammed in Abuja says the shooting took place at a two-storey building which has just been built but is not yet complete.
Young men have been using the building to sleep in at night, he says.
Boko Haram is most active in north-eastern Nigeria, where a state of emergency was imposed in May.
If confirmed, it would be the first time Boko Haram has staged an attack in Abuja this year.
Attacks in the north-east have increased recently despite a massive military deployment to the worst-affected areas.
In the latest incident in Borno state, officials said at least 87 people had been killed by militants, who disguised themselves in military uniforms at a checkpoint outside the town of Benisheik. They shot dead those trying to flee.
The group wants to create an Islamic state across Nigeria and has waged a deadly insurgency since 2009.
'Digging for arms'
The security team which approached the building were acting on information received from two men, agents said.
"No sooner had the team commenced digging for the arms, than they came under heavy gunfire attack by other Boko Haram elements," Reuters news agency quotes a statement from State Security Service as saying.
Our reporter says the building is in Abuja's Apo district, home to a huge residential complex for Nigerian parliamentarians.
Abuja suffered two major Boko Haram attack two years ago, when a suicide bomber rammed a car into the police headquarters, killing eight people in June 2011.
About two months after that, the group attacked the UN headquarters in Abuja, killing 23 people.
The attack near Benisheik took place on Tuesday, but news of it was slow to emerge as all phone lines have been cut off in an effort to help the military offensive.
The Boko Haram members drove into the town in about 20 pick-up trucks, the AFP news agency quoted an anonymous security source as saying.
The BBC's Nigeria correspondent Will Ross says it was one of the deadliest since the state of emergency was declared.
In the three days since the attack, health workers have been loading dead bodies onto trucks and some reports say the militants killed more than 140 people.
"Apart from the dead bodies recovered today [Thursday], we collected 55 on Wednesday and the fact is that we did not go deep into the bush where I strongly believe that many people have fallen there," Nigeria's Daily Trust newspaper quotes Abdulaziz Kolomi, an official with state's environmental protection agency, as saying.
There was also an attack by suspected militants on Wednesday night in neighbouring Yobe state, which is also under a state of emergency but has not witnessed so much violence.
A resident of Buni Yadi told the BBC Hausa Service that Islamists attacked the town at about 22:30, burning the police station and other public buildings.
"A soldier was killed in a shootout and the wife of the [divisional police chief] was burnt to death in her home," state police commissioner Sanusi Rufa'i told AFP.
Local vigilante groups have been formed to help counter the militants but scores of these volunteers have been killed in recent weeks.
Last month, the army said it had killed Boko Haram's leader Abubakar Shekau but this has not been confirmed and the militants' attacks have continued.