A Brazilian gamer once asked me if ECLs have any explanation in-game for what they are and how they work, or was it something purely for balance that existed only between players and DMs. This is the answer I gave him.
Think of it this way:
A group of 1st-level human characters facing a group of 4 orcs (total Encounter Level of 2) are in for a tough fight; the orcs outmatch them (because the average character level is 1 and the EL for the encounter is 2). The humans are taking a great risk in fighting the orcs; the humans have a significant chance of losing the fight.
A group of 1st-level drow characters (each with total ECL of 3 ... 1 for class level, 2 for level adjustment) facing a group of 4 orcs are in for an easy fight; the orcs are weaker than them (because the average effective character level is 3 and the orcs' EL for the encounter is 2). The drow aren't really taking a risk; they have almost no chance of losing the fight.
The drow have it easier than the humans do from day one of their adventuring career. They don't have to work as hard as humans do to succeed. Because they don't have to work as hard, they don't work as hard. This is reflected in game terms by the drow getting fewer XP than the humans for an encounter of the same EL (because 3rd-level characters get fewer XP for defeating a creature than 1st-level characters do*). So the drow aren't earning as much XP because they are getting "easy fights" compared to the humans. They're taking less of a risk, and aren't being rewarded as much.
(* OK, technically they do get the same XP because the XP awards are the same for levels 1-3, but at higher levels it is actually different. In any case, the amount of XP awarded does less for a higher-level character because more XP is needed to reach the next level.)
So, in the terms of the game world, the drow can afford to be more lazy, and so they don't climb levels as fast as humans do, who don't have the luxury of being lazy (because humans don't have SR, great ability score bonuses, and so on).