Fighting a dagger spec you either have to:

A) Have the gear to cover the stat loss. At hero this is usually >+15 str/10 dex. It's possible, but you have to have a pretty a nice set. However, if you do have such a set, you essentially render the dagger spec obsolete.

B) Have the burst damage to nullify a dagger spec in 3-4 rounds. If you don't have the gear to cover the stat loss, then aganist a dagger spec you've got about 4-5 combat rounds before you're in danger of being 1-2 rounded due to maledicts adding up. So you have to be able to gauge whether or not you can burst him down in that time. For instance, if I was a giant sword spec I might throw out a doublethrust right when the fight starts. If I chunk him down from perfect health to bleeding wounds or so, then I can think about trying to flurry to end this thing, or maybe go for a bash - melee hit or two - 2nd doublethrust to end things.

Caveat: Hand spec as first spec is rough in general and not what I would choose to fight dagger specs. The only good side is that you don't have to worry about dropping your weapon. You don't get much in the way of weapon avg (unless felar) or any +hit/dam/str from your weapons. That hurts until hero when things can kind of even out.

Caveat 2: If I had to choose a build to beat elf dagger spec, it'd probably be some combination of sword/pole/mace. Pole defends REALLY well against dagger. Even as a giant, pre STSF you can usually straight up bash/chop/bash/chop a dagger spec down with a polearm provided you have the +str. Sword spec also kind of trumps dagger if you can handle/burst past the maledicts. Mace has boneshatter for a hefty maledict, which against typical dagger specs (arial/elf) if they don't have much in the way of +dex can really fuck them, plus the burst damage of drum.