Transport barriers in the ocean influence the mixing of heat, salinity, debris, and the movement of complex mobile ecosystems. While their non-stationary boundaries can now be identified in a mathematically rigorous fashion, this identification relies on the availability of surface velocity fields at suitable temporal and spatial resolutions. A wealth of independent trajectory and velocity information from ocean drifters is also available but has remained largely unexploited in detecting transport barriers. The main difficulty in such a detection is the sparsity and intermittency of drifter data, which prevents constructing a sufficiently detailed drifter velocity field. Here we discuss how recent developments with deep neural networks enable the construction of such a drifter velocity field from observed drifter trajectories. This approach shows clear potential for uncovering the location of material transport barriers, such as fronts, eddies, and jets, from available drifter data, as well as for improving Lagrangian drifter models.