Summary

Cosyr is a particle beam dynamics simulation code with multi-dimensional synchrotron radiation effects. It tackles a fundamental problem of the self-consistent nonlinear dynamics of a particle beam from its complete self-fields, particularly the radiation fields, i.e the coherent synchrotron radiation (CSR) problem. The latter underpins many accelerator design issues in ultra-bright beam applications, as well as those arising in the development of advanced accelerators. It is is actively developed and supports two levels of parallelism with MPI and Kokkos. A related poster was presented at the IPAC'21 conference.

Algorithm

It is based on the Lienard-Wiechert field formula of the retarded Green’s function for the Maxwell equations. A local cartesian moving mesh is used to follow the trajectory of the beam. For each particle, their fields (or potential) are then calculated on the radiation wavefronts that intersect with the moving mesh. These wavefronts are emitted along the trajectory of the particle at a specified interval. Their discretization points (called wavelets) form another mesh that is naturally adapted to the emission. In particular, these wavefronts are divided into two groups depending on the retarded time:

• dynamic: they correspond to the wavefronts associated to a large retarded time (longer than the typical time scale of the beam evolution for instance). Here, the fields are calculated by taking into account the full dependence on the position, velocity and acceleration at emission.
• subcycle: they correspond to the wavefronts that are emitted with a retarded time shorter than the beam evolution time scale. Here, they are treated as if being emitted from subcycle time steps where the emitting particles are assumed to follow the reference trajectory but shifted by their offset from the reference particle. As such, the subcyle wavelets and their fields are precalculated using the reference particle which has the reference energy and experiences only external fields.

The wavelet fields are then interpolated to the moving mesh, and the contributions from all particles are summed together to obtain the beam self-fields. The beam fields are again interpolated to the particles and are used to push them such as in a conventional PIC method. The algorithm is depicted in the snippet below:

initialization  
for each time step {  
  push reference particle  
  update moving mesh  
  push other particles  
  if (emit wavelets) {  
    update emission data  
    for each particle {  
      shift subcycle wavelets
      emit dynamic wavefront
      for each wavefront {
        if wavefront intersects mesh {
          compute wavelets fields
        }
      }
    }
    interpolate fields to mesh
  } 
}

By doing so, the high frequency radiations can be separated from those that can be resolved on the moving mesh by selecting the wavelets on a given wavefront. In addition, the synchrotron radiation can be emitted from the trajectories of the particles, which dynamically respond to the self-fields. Finally, the algorithm provides multiple levels of parallelism to leverage high performance computing clusters. It is more computationally intensive than other CSR models though. As such, it is currently limited to low energy beams.

Performance

Cosyr is designed to leverage multi-GPU nodes. It exploits the fact that the field kernel, wavelets emission and particle update are completely local. For a given beam, a subset of particles is assigned to a MPI rank. The field kernel is primarily parallelized over particles to be run on manycore nodes or GPU. The wavelet-to-mesh interpolation is done in a multithreaded way using Portage.

Build and use

It can be built on Linux or macOS using CMake.
It requires a C++14 compiler endowed with OpenMP.
The field kernels are designed to be run on manycore CPU or GPU.
For optimized builds on Intel KNL, we recommend the Intel compiler.
To run on Nvidia's GPU, it should be compiled with nvcc.

It was not currently tested on AMD's GPU.

Dependencies

• kokkos for the wavefronts and field kernels.
• cabana for the particle pusher.
• portage for the interpolation step.
• pybind and a python environment to handle simulation parameters.

The instructions to build the project and its dependencies can be found here.

Running

The simulation parameters are given through an input deck.
The latter is simply a regular Python script.
Those parameters are loaded into the Python locals and can changed as needed.
A set of examples are given in the input directory.
It is also possible to initialize a beam array for the simulation as shown here.
To run: mpirun -np 4 ./cosyr params.py

License and contributions

Cosyr is developed at Los Alamos National Laboratory (C20129).
It is supported by the Laboratory Directed Research and Development program (LDRD).
It is open source under the BSD-3 licence.
It is developed by:

• Chengkun Huang
• Feiyu Li
• Hoby Rakotoarivelo
• Boqian Shen

Other contributors:

• Rao Garimella
• Thomas Kwan
• Bruce Carlsten
• Robert Robey
• Orion Yeung
• Parker Pombrio

Copyright

© 2021. Triad National Security, LLC. All rights reserved. This program was produced under U.S. Government contract 89233218CNA000001 for Los Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC for the U.S. Department of Energy/National Nuclear Security Administration. All rights in the program are reserved by Triad National Security, LLC, and the U.S. Department of Energy/National Nuclear Security Administration. The Government is granted for itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide license in this material to reproduce, prepare derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so.