Abstract: Fuel assemblies are the central components of a reactor. The core fuel pellets in the fuel pins will swell and deform and the fuel cladding may even break under the complex environment of high temperature, high pressure and intense neutron radiation field, which threats the safety of the reactor. To better understand the changes in the behavior of the fuel assembly in the reactor and study the central void formations and deformations of fuel pins in fuel assemblies to high burn-up, high-energy X-ray non-destructive testing is an effective technical means. Irradiated nuclear fuel assembly has a strong radioactivity, it is necessary to optimize the design of the detector system and the collimator to reduce the effect from gamma rays emitted from the irradiated fuel assembly during detection system designing phase. Through modeling, estimating and optimization, the optimal size of the detector unit is obtained and the collimator design is optimized which can lay the foundation to improve the quality of the reconstructed images of the fuel assembly nondestructive system.