Instead of determining it from the pointer type, explicitly pass the type to load.
Please read the rustc-dev-guide chapter on Backend Agnostic Codegen.