Past instructor of CSCE 121 and former Aggie here. This is a common point of confusion; you're not alone. I think multiple perspectives are helpful towards understanding, so I'll share. I'm imperfect, but here's what I've found helps; how I introduce these concepts.
It is particularly important to recognize that the context in which `*` and `&` present makes all the difference. You might see `*` and `&` present as declarator operator in a declaration of a local variable or function parameter (the semantics of argument passing are the same of initialization). As a declarator operator, you're creating a pointer _object_ or binding a name (creating a reference) to an existing object. Contrast this, when you're using `*` and `&` in an expression (and not as declarator operator), you've got the dereference operator and the address of operator respectively.
References and argument passing
I like to think of references as a nickname/alias for an existing object. It is important to recognize that an object may take on different names in different scopes (for instance, when an object is passed by reference). Here's an imperfect example that one might relate to: one might different names in different circles. For instance, my friends from graduate school call me "Michael", friends from college "Mike", though these names refer to the same individual (i.e., me!) when used in the proper context.
You can think of pointers as simply compound objects built from the primitive types/UDTs. Pointers are an object in their own right; their value is an address to an object of their base type. This allows indirect access to the object: using the address stored in a pointer, you can go out to that address, and begin interpreting the bits there according to the pointer's base type. Here's another imperfect example: some of my friends have my address in their contact list. Those friends know where I live, so if they want to interact with me, they can come over. Notice: they have to look at the address (pointer) and then walk to that address (dereference operation) in order to interact with me. Many friends can have my address: it can reside in more than one person's contact list. Those friends can look at my address in their contact list, without coming over; they can even update my address should I move.
"pointers" become inherently complicated when dynamically allocated objects are introduced. This is where memory leaks, dangling pointers, etc. emerge. The reality is that pointers do not necessarily become more "complicated" at this point, but it is the management of those dynamically allocated objects pointed to that makes things "complicated".
To understand the need for dynamically allocated objects, you have to understand the different regions of memory available to you and the implications of putting an object in one place, and not another. Functions and the stack. Allocation of memory to variables on the stack. Dynamically allocated objects.
Hopefully this helps some.