I can't calculate how many spikes were in there.
Probably as many as it takes to kill any one of the Bristlebacks times the number of BB's, as they wouldn't heal if they're dead.
If they can't survive from Octarine's lifesteal, it'll happen when they take enough damage to drop to 100% hp. This is the most likely scenario.
In case they can post-heal out of death (a la "bloodseeker blademail vs axe" bug scenario - blood reaches 0 hp, but survives culling blade thanks to the heal from killing axe) due to the proximity of the damage they take and the revenge quill lifesteal (if it's under a server tick), things get harder. Then it's BB's Health + the amount of hp they heal off of attacking the other bristlebacks. To simplify the matter, at any one point, we can assume that all bristlebacks are dealing and taking the same amount of damage, a simplification but it effectively works as they all start and die together anyways.
Continuing the post-damage healing assumption. BB will "overheal" from using the revenge quills (he does 125% lifesteal in damage off spikes due to 5 targets) and I doubt you can heal past 100% hp, so I would assume they all die when the damage component they take exceeds 200% of their hp, as the post-damage heal won't be able to save Bristleback's life in that tick. Bristleback does 200% of his hp in quill lifesteal when he does 160% of his own HP in damage, so I could assume, they die at the point when they thrown enough Quills at once that they would do over 160% of their own Hp in a single salvo of damage.
Now a bristleback with 5 hearts and 1 Octarine on level 25 has 7686 health, 12 armor, which should be 13220 EHP. The bristleback 40% damage reduction bumps it EHP up another 66.65% if Dota 2 wiki isn't lying to me, so I'll have his final HP as 22031 HP.
To do that much damage in one single salvo of Quill spray, Bristle must attack someone with 610 stacks of Quill Spray, which would mean every Bristle threw out 122 Quills and then died.
If you must indeed do the "overheal" damage, then Bristleback must do 35250 damage to get a kill in one Quill Spray. THis achieves a total of 977 Quill Spray stacks before death, meaning each Bristleback throws out 195 Quills, then two bristles throw an extra one and everyone's dead.
But I have my doubts any of this is sane.