MySQL: Select all that only have two rows, with specific values?

MySQL: Select all that only have two rows, with specific values?

I would like to grab all the users that ONLY have two roles, which are 1
and 4.
One user role is stored like this:
user_id role_id
54321 1
54321 4
54322 1
54323 1
How can i make a query, that grabs the user_id 54321, because it Only have
two roles and these two are 1 and 4?
I can use WHERE role_id IN (1, 4) but this will also grab users that have
other roles.