You may be thinking of Air Gear (2006) – Ikki uses motorized inline skates (Air Treks) and enters the "Trophaeum" tower/alternate battles.
Or Rurouni Kenshin – Kenshin's Hiten Mitsurugi-ryū speed resembles rollerblading, but no alternate world entry.
Other candidates:
Air Gear best fits "roller blades + alternate world/competition." Provide more details (art style, year, plot) to confirm.