You see it a lot on YouTube, forums, articles. From graphics to library to controller the 64 has a bad rep in the community. Games like Mario 64, OOT, GoldenEye are slammed for being dated or flat out bad games that everyone was delusional about. Look at PS1 and Saturn though and those same critiques aren't ever given generally. Before diving into this please know: I love all three, and brand identity only hurts the enjoyment of video games in general. These consoles have come and gone, we're left with thousands of interesting games to enjoy, hate, study, and laugh at. This goes for every console gen. Without further a do lets go!
Regarding library yes, 64 doesn't offer as large a number or as much variety. The above mentioned games, however, seem to receive critiques you don't find on the other consoles. Controls, camera and quality of game design. When you look at 3D games on the competition you find a ton of the same issues that go ignored. Tomb Raider, Burning Rangers, Syphon Filter among others have just as many "dated" issues but you never see them judged for that fact. One positive you never hear is that 64 games generally didn't use tank controls, instead opting for instantaneous 3D movement due to the analogue stick. This kind of movement has technically aged better as every game nowadays controls this way and tank controls are dead for obvious reasons.
For graphics you hear how the 64 produced ugly muddy textures and that the competition's graphical output is much better. I think this is unfair given PS1 and Saturn's graphical issues. PS1 warping, jaggies on both consoles among others. All three are victims of early 3D hardware and have all aged in different ways. I fail to see how 64 necessarily looks any "worse" than the competition. Perfect Dark, Conker and Castlevania are examples of the 64's strengths in producing varied detailed environments on top of the 64's anti-aliasing. Put Medal of Honor next to Perfect Dark and it's no competition which one looks better. Cruisin' USA vs. Daytona is another good example. And again this isn't a dick measuring contest, it's a common hypocritical and unrealistic complaint levied at the 64. All three consoles offer the charm of early 3D in their own unique ways and neither really looks better than the other as an absolute. A friend of mine says PS1 and Saturn games "look so much better" than anything on the 64, and I genuinely don't see how they're any better really. It's all about the games and how devs utilized what they were working with.
Then the controller. Nobody can judge someone for not liking a controller but it's never about disliking the controller. I've seen "HOW ARE YOU SPOSED TO HOLD THIS THING??" so often by people who are very much in the retro community. You'd think these players would understand the controller's most common utilization of the middle and right grips. It is a strong default controller for FPS games, providing an almost mouse and keyboard feel. PS1 and Saturn required the Dualshock and 3D pad and respectively for FPS games to play better. Even then not all FPSs on those platforms support those controllers. On 64 you're guaranteed more natural aiming and moving out of the box, but again this is a strength you don't often see. Not saying its an amazing controller, and yeah it breaks… a lot. Thing is for 3D gaming you have to give one to the 64 for giving true 3D control by default with the system.
Bottom line? I think the PS1 and Saturn don't face criticisms you see for the 64 on the same issues. Early 3D games in general haven't aged well. You have your outliers like Crash but every console's 3D games faced poor cameras, clunkier controls and dated graphics. There's a charm in the aesthetic, as there is with pixelart, and if anything we should consider the time period the hardware and games were made in. If everyone wants competition then each console has one-ups on their competition. Let's all remember to enjoy the history of gaming and keep in mind the context in which this was all released.