Can you go over your Manager's head and talk to his Manager about the problem? Sounds like your Manager is not pulling his weight on the project. You should definitely escalate this. Anyone on the project that you can talk to, like a peer of your Manager?