Quantum photonic neural networks are brain-inspired, nonlinear photonic circuits that can learn to tackle many key challenges emerging quantum technologies face. By combining the strengths of artificial intelligence and photonic integrated circuits, these networks can learn to perform near-deterministic, high fidelity (i.e., near 100% chance of success) quantum processing as would be necessary for fundamental elements of the future quantum internet, for example. However, proposals of the network to date have assumed that all of its components work perfectly. Instead, we developed advanced simulations to model and analyze the performance of imperfect quantum photonic neural networks based on current state-of-the-art experimental capabilities, including the relevant imperfections. In doing so, we found that imperfect networks can still achieve near-deterministic operation and unraveled the intricate relationship between chip fabrication imperfections, weak optical nonlinearities, and network size, thus providing a guide to the optimal design of future quantum photonic neural networks for experimentally viable settings.